Testing Non-Deterministic AI: How Many Runs Are Enough?
qabash.comAmbassador
Nov 23, 2026
Quick answer
To test non-deterministic AI, run each case many times and treat the result as a pass rate with a confidence interval, not a single verdict. Gate releases on the interval's lower bound (for example, "we're 95% confident the pass rate is at least 90%"). If you need near-zero failures, use the rule of three: zero failures in 300 runs means the failure rate is below 1% at about 95% confidence. When comparing two prompts or models, use a two-proportion test before believing a difference.

Your LLM test passed 9 times out of 10. Is the feature 90% reliable?
You don’t know. With 10 runs, the true pass rate could reasonably be anywhere from about 60% to 98%. That’s the difference between “ship it” and “this fails for four users in ten.”
Classic test automation assumes one run gives one verdict. LLM features break that assumption, and most teams respond by either ignoring the randomness or running everything three times and hoping. This guide covers testing non-deterministic AI with three small statistical tools any tester can use, and code you can paste into a pytest suite today.
Why are AI features non-deterministic?
A language model generates text by sampling: at each step it picks the next token from a probability distribution. Temperature, top-p and similar settings control how much randomness goes into that choice.
Setting temperature to 0 reduces variation but usually doesn’t eliminate it. Batching on the provider’s servers, floating-point differences across hardware and silent model updates can all change the output for an identical request. Agents amplify this, because one different word early on can lead to a different tool call and a different path.
So “run it again and see” isn’t a bug in your process. It’s the nature of the system. The fix is to make “run it again” rigorous.
From pass/fail to pass rate
For a deterministic test, one run is a fact. For an LLM test, one run is a sample. The right unit is a pass rate over N runs, for each important case:
- Run each evaluation case N times with production settings.
- Count passes using a fixed oracle (exact checks, schema checks or a calibrated judge).
- Report passes/N, plus how confident you are in that rate.
That last step is the one teams skip, and it’s where the wrong decisions come from.
Tool 1: confidence intervals (how sure are we?)
A confidence interval says: “given what we observed, the true pass rate is probably in this range.” For pass/fail data, the Wilson score interval is the practical choice. Unlike the textbook formula, it behaves well with small samples and with rates near 0% or 100%, which is exactly where test data lives.
# llmstats.py: the three numbers you need to test a non-deterministic feature
import math
def wilson_interval(passes: int, runs: int, z: float = 1.96) -> tuple[float, float]:
"""95% confidence interval for a pass rate (Wilson score; safe near 0 and 1)."""
p = passes / runs
centre = (p + z * z / (2 * runs)) / (1 + z * z / runs)
half = z * math.sqrt(p * (1 - p) / runs + z * z / (4 * runs * runs)) / (1 + z * z / runs)
return max(0.0, centre - half), min(1.0, centre + half)
def runs_for_zero_failures(max_failure_rate: float) -> int:
"""Runs needed, all passing, to claim failure rate < max at ~95% (rule of three)."""
return math.ceil(3 / max_failure_rate)
def two_proportion_p_value(pass_a: int, runs_a: int, pass_b: int, runs_b: int) -> float:
"""Two-sided p-value: is prompt B's pass rate really different from prompt A's?"""
pooled = (pass_a + pass_b) / (runs_a + runs_b)
se = math.sqrt(pooled * (1 - pooled) * (1 / runs_a + 1 / runs_b))
if se == 0:
return 1.0
z = (pass_b / runs_b - pass_a / runs_a) / se
return math.erfc(abs(z) / math.sqrt(2)) # two-sided normal tail
Here is the same observed 90% pass rate at different sample sizes, computed with wilson_interval:

Ten runs barely tell you anything. Going from 100 to 300 runs narrows the range much less than going from 10 to 30. Precision grows with the square root of N, so each extra digit of confidence costs a lot more runs.
Trade-off: runs cost tokens and time. Spend them where the risk is. Fifty runs on the refund-policy answer, ten on the greeting.
Tool 2: the rule of three (how rare are failures?)
Some behaviours must essentially never fail: leaking another user’s data, calling a destructive tool, ignoring a safety rule. Your test passes every run. How rare could the failure still be?
The rule of three answers that: if you see zero failures in N runs, the true failure rate is below about 3/N at 95% confidence.
| Zero failures in | Failure rate is probably below |
|---|---|
| 30 runs | 10% |
| 100 runs | 3% |
| 300 runs | 1% |
| 3,000 runs | 0.1% |
This is sobering, and useful. A guardrail test that passes 20 times in a row tells you its failure rate is probably below 15%. That isn’t a safety claim. It’s also why hard guardrails belong in code (allowlists, permission checks) rather than in the prompt: code doesn’t need 3,000 runs to trust.
Tool 3: comparing two prompts (is it better or just noise?)
Someone changes the prompt. Old version: 41/50 passes. New version: 45/50. The PR says “improves accuracy from 82% to 90%.”
two_proportion_p_value(41, 50, 45, 50) # 0.249
A p-value of 0.25 means a difference this size would show up about one time in four from randomness alone. That’s not evidence of improvement. With 200 runs each (160 vs 180, the same 80% vs 90%), the p-value drops to about 0.005, and now the improvement is believable.
Two practical rules:
- Compare on the same cases. Run both prompts against the same evaluation set, so case difficulty isn’t the difference.
- Decide the sample size before you look. Running “until it looks better” and stopping is the fastest way to ship noise.
A release gate in pytest
The gate uses the interval’s lower bound, not the raw rate. “Observed 95%” isn’t the claim; “at least 90%, with 95% confidence” is.
# test_llmstats.py: a release gate built on the lower bound, not the raw rate
from llmstats import runs_for_zero_failures, two_proportion_p_value, wilson_interval
def test_small_samples_are_wide():
low, high = wilson_interval(9, 10) # "90% pass rate" from 10 runs
assert round(low, 2) == 0.60 and round(high, 2) == 0.98
def test_release_gate_uses_lower_bound():
low, _ = wilson_interval(190, 200) # 95% observed
assert low >= 0.90 # gate: we are confident the true rate is at least 90%
def test_rule_of_three():
assert runs_for_zero_failures(0.01) == 300 # 0 failures in 300 runs => <1% at ~95%
def test_prompt_change_that_is_just_noise():
# Prompt A: 41/50, prompt B: 45/50. Looks better; is it?
assert two_proportion_p_value(41, 50, 45, 50) > 0.05
pip install pytest
pytest -q
# 4 passed in 0.08s
In a real suite, passes and runs come from your evaluation job: loop over cases, call the model N times, score each output with your oracle, then assert on wilson_interval(passes, runs)[0].
How many runs do you need?

Cost control tips:
- Split the suite. Deterministic code tests (parsing, tools, guardrails) run on every commit with a fake model. Statistical runs happen nightly or on prompt and model changes.
- Cache by input and settings. Re-running an unchanged case against an unchanged model wastes money.
- Escalate only borderline cases. If 10 runs show 10/10 or 2/10, you may not need 100. Spend more runs where the interval straddles your threshold, and fix that rule in advance.
Common mistakes testers make
| Mistake | Why it happens | Better approach |
|---|---|---|
| One run per test | Habit from deterministic suites | Pass rate over N runs for important cases |
| Re-running until green | Feels like handling flakiness | Fixed N; report the rate, not the last result |
| Reporting the raw rate | Simple and looks precise | Report the interval; gate on its lower bound |
| Trusting small A/B wins | The new number is higher | Two-proportion test, larger samples |
| Same budget for every case | Easy to configure | Size runs by risk |
| Guardrails proven by a few passes | Green feels safe | Rule of three, and move hard rules into code |
Interview questions
Beginner
Why does the same LLM prompt give different outputs?
Sampling randomness, plus infrastructure effects such as batching even at temperature 0.
What is a pass rate?
Passes divided by runs for a test case run repeatedly.
What does a confidence interval tell you?
The range the true pass rate probably lies in, given the runs observed.
Why isn’t 9/10 the same as 90%?
With 10 runs the plausible range is roughly 60% to 98%.
Intermediate
Why use the Wilson interval?
It stays accurate for small samples and rates near 0 or 1, unlike the simple normal approximation.
What is the rule of three?
Zero failures in N runs means the failure rate is below about 3/N at 95% confidence.
How do you compare two prompts?
Run both on the same cases with a fixed sample size and use a two-proportion test.
Why gate on the lower bound?
It’s the pass rate you can defend, not the one you happened to observe.
Advanced
How would you budget runs for 200 evaluation cases?
Tier by risk, use small N for low-risk cases, escalate borderline ones with a pre-set rule.
What goes wrong if you run until a result looks good?
Optional stopping inflates false positives; decide N in advance.
How do you test a guardrail that must never fail?
Enforce it in code with deterministic tests, and use the rule of three only to bound the model-side behaviour.
How does this relate to pass^k for agents?
pass^k asks whether all k runs succeed, a stricter reliability measure; intervals tell you how precisely you’ve measured either one.
How AI changes this
This is a skill AI created demand for. Testers who were happy never computing a confidence interval now need one to answer “is the new prompt better?”
AI also helps. Assistants can write the evaluation loop, compute intervals and spot when a comparison is underpowered. Keep the decision rules (thresholds, sample sizes, what counts as a pass) human-owned and written down before the run.
Into 2027, expect release gates on AI features to look more like A/B experimentation than unit testing: pre-registered thresholds, run budgets and dashboards of pass-rate trends across model versions. Teams that already think in intervals will adapt fast. A bug found in production is expensive. A bug prevented is priceless.
Best practices checklist
- Run important cases repeatedly with production settings
- Report pass rates with a confidence interval
- Gate releases on the interval’s lower bound
- Use the rule of three for “must never fail” claims
- Compare prompts on the same cases with a two-proportion test
- Fix sample sizes and thresholds before running
- Size run budgets by risk
- Keep hard guardrails in code with deterministic tests
Conclusion
Testing non-deterministic AI doesn’t need a statistics degree. It needs three habits: report pass rates with intervals, use the rule of three for rare failures, and test differences before believing them.
The payoff is a release meeting where “it passed” means something. Next time someone says the new prompt is better, ask how many runs, and what the lower bound was.
Automation can execute tests. Thinking still belongs to testers.
Keep testing. Keep learning. Keep questioning.
Go deeper · QA Bash kit
AI for QA Engineers: Getting Started
Use AI tools in your testing work, with prompts that work and checks that keep you safe.
0–1 yrs · 10 testing workflows
₹299₹374
Rate this article
8.5/10 average · 25 ratings
Discussion
Be the first to weigh in
Your question or experience helps the next tester who reads this. Sign in and earn +25 XP for every approved comment or reply.
Written by
Frequently asked questions.
Keep reading




