Why Testing AI Is Different
Non-Determinism — Designing Tests for Systems That Don't Give the Same Answer Twice
Non-Determinism — Designing Tests for Systems That Don't Give the Same Answer Twice
Every testing habit you've built so far assumes a foundational premise: given the same input, the system produces the same output. Traditional test assertions — assertEquals(expected, actual) — depend on it. LLM-powered systems break that premise on purpose. Sampling temperature, token-level randomness, and even provider-side infrastructure changes mean the same prompt can produce meaningfully different (but individually valid) responses on consecutive calls.
Where the Non-Determinism Comes From
Most LLM APIs expose a temperature parameter that controls how much randomness is injected into token selection. At temperature=0 many providers are "mostly" deterministic but not guaranteed to be — floating-point non-associativity in batched GPU inference and backend load-balancing across slightly different model shards can still produce different outputs for identical requests. Anything above temperature=0 is deliberately stochastic by design.
Why assertEquals Fails You Here
If you ask an LLM "summarize this support ticket in one sentence" ten times, you'll likely get ten different (but each reasonable) sentences. An exact-match assertion will fail nine times even though the system is working correctly. Testing AI applications means shifting from exact-match testing to property-based testing: instead of asserting what the output is, you assert properties the output must have.
| Traditional assertion | AI-appropriate assertion |
|---|---|
output === "The user requested a refund." | output contains a reference to a refund request |
output.length === 42 | output.length is between 10 and 100 words |
response.status === "approved" | response.status is one of a valid enum, and the reasoning is internally consistent |
Three Practical Strategies
- Constrain the non-determinism where you can. Set
temperature=0(or the lowest your use case tolerates) for anything you plan to test deterministically — classification, extraction, routing decisions. Save higher temperature for genuinely creative tasks you won't unit-test the same way. - Test properties, not strings. Assert on structure (valid JSON, required fields present), constraints (word count, forbidden phrases absent), and semantic similarity (embedding distance to a reference answer above a threshold) rather than exact text.
- Run N samples, assert on the distribution. For genuinely stochastic behavior, running the same prompt 5-10 times and asserting that e.g. 90% of outputs satisfy a property is far more honest than a single pass/fail run — and it catches "sometimes wrong" failure modes single-shot testing misses entirely.
The Mindset Shift
You are not testing a function anymore — you are testing a policy: a system that behaves within a distribution of acceptable outputs. Every lesson in this path builds on that one idea.
💬 Discussion
Think of a feature you've tested that had even a small amount of non-determinism (retry logic, random IDs, timing). How did your team handle it, and how would that approach need to change for an LLM that's non-deterministic by design rather than by accident?
Why does `assertEquals(expected, actual)` typically fail as a testing strategy for LLM output, even when the system is working correctly?
Next Lesson
The New Test Pyramid for AI Applications