LLM Evaluation and LLM-as-a-Judge: A Practical Guide
qabash.comAmbassador
Nov 19, 2026
Quick answer
LLM evaluation measures the quality of an LLM system's output against defined criteria across a dataset of cases, instead of asserting exact strings. Evaluate in four layers: (1) deterministic contract checks on structure, (2) a golden dataset of required and forbidden facts, (3) pass-rate thresholds over repeated runs, and (4) an LLM-as-a-judge for qualities like faithfulness. Before trusting the judge, calibrate it: measure its agreement with human labels (using Cohen's kappa, not raw agreement) and check that swapping answer order doesn't change its verdict.

Your first LLM test usually looks like this: send a prompt, compare the reply to an expected string, and watch it fail the next morning with a perfectly good answer worded differently.
That’s not a flaky test. It’s the wrong kind of test. LLM evaluation starts with accepting that you can’t assert exact output. You measure properties of the output over many runs, and for the qualities code can’t check, you use an LLM-as-a-judge that you’ve tested first.
This guide gives you four evaluation layers, a judge calibration method, runnable pytest code for all of it, and the trade-offs that decide which layer catches which bug.
If you want the broader map of LLM testing types first (functional, safety, bias, performance), start with our LLM testing overview. This guide goes deep on one part of it: evaluating output quality.
Why does LLM evaluation need a different approach?
Traditional tests assume one input gives one output. LLMs break that assumption in three ways:
- Non-determinism. The same prompt can give different wording, ordering, or even facts. Lowering temperature reduces variation but doesn’t remove it.
- Open-ended correctness. “Refunds are available for 30 days” and “You have a month to request a refund” are both right. String equality calls one of them a failure.
- Silent drift. Your provider updates the model, you tweak a prompt, or someone re-indexes the knowledge base. Nothing in your code changed, and the behaviour still did.
The practical consequence: an LLM test is closer to a statistical quality check than to a unit test. You still write unit tests, just around the model rather than on the exact text it produces.
Automation can execute tests. Thinking still belongs to testers, and with LLMs most of that thinking goes into deciding what “correct” means.
What should LLM evaluation actually measure?
Measure things that stay true no matter how the model phrases its answer:
| Check this | Not this | Why |
|---|---|---|
| Output parses as valid JSON with required keys | Exact JSON string | Key order and whitespace vary |
| Answer contains “30 days” | Answer equals a full sentence | Facts are stable; wording isn’t |
| Answer never contains “60 days” | Nothing about wrong facts | Forbidden-content checks catch hallucinations |
| Cited sources ⊆ retrieved documents | Number of sources | A citation to an unretrieved doc is fabricated |
| ≥ 80% of 50 runs pass | 1 run passes | One sample tells you nothing about a distribution |
| Judge score ≥ 4/5 on faithfulness | “Sounds good to me” | Makes subjective review repeatable |
The design rule: prefer cheap deterministic checks, and only escalate to an LLM judge for qualities you genuinely can’t check in code. Every judge call adds cost, latency and its own error rate.
The four layers of LLM evaluation (with code)
We’ll evaluate a small RAG-style support bot. It retrieves documents, asks the model to answer only from them, and requires JSON back.
The key design choice is that the model is injected as a plain function. Tests pass a fake, the eval job passes a real client, and the app code doesn’t care which it gets.
# support_bot.py — the system under test: a RAG-style support bot.
import json
from typing import Callable
# Any provider fits behind this signature: prompt in, text out.
# Tests inject a fake; production injects a real API client.
ModelFn = Callable[[str], str]
KNOWLEDGE_BASE = {
"refund-policy": "Refunds are available within 30 days of purchase.",
"shipping": "Standard shipping takes 5 to 7 business days.",
"warranty": "Hardware carries a 1-year limited warranty.",
}
def retrieve(question: str) -> dict[str, str]:
"""Naive keyword retrieval. Real systems use embeddings; the test shape is the same."""
q = question.lower()
return {doc_id: text for doc_id, text in KNOWLEDGE_BASE.items()
if any(word in q for word in doc_id.split("-"))}
def build_prompt(question: str, context: dict[str, str]) -> str:
docs = "\n".join(f"[{doc_id}] {text}" for doc_id, text in context.items())
return (
"Answer ONLY from the documents below. If they don't cover it, say you don't know.\n"
'Reply as JSON: {"answer": str, "sources": [doc_id, ...]}\n\n'
f"Documents:\n{docs or '(none)'}\n\nQuestion: {question}"
)
def answer(question: str, model: ModelFn) -> dict:
context = retrieve(question)
raw = model(build_prompt(question, context))
data = json.loads(raw) # Raises on non-JSON output; the contract test relies on this.
return {"answer": data["answer"], "sources": data["sources"], "retrieved": list(context)}
Now the tests. Save this as test_support_bot.py next to it and run pytest -q. All seven tests pass offline in milliseconds.
Layer 1: Contract checks
These check structure, not meaning. They’re deterministic, so they belong on every commit.
# test_support_bot.py — run with: pytest -q
import json
import random
import pytest
from support_bot import answer
# ── A fake model so the suite runs offline, fast, and free ──────────────
def fake_model(prompt: str) -> str:
"""Stands in for a real LLM. Swap for your provider call in the eval job."""
if "[refund-policy]" in prompt:
return json.dumps({"answer": "You can get a refund within 30 days.",
"sources": ["refund-policy"]})
if "[shipping]" in prompt:
return json.dumps({"answer": "Shipping takes 5 to 7 business days.",
"sources": ["shipping"]})
return json.dumps({"answer": "I don't know based on our documents.", "sources": []})
# ── Layer 1: contract tests (deterministic, run on every commit) ────────
def test_output_matches_contract():
result = answer("What is your refund policy?", fake_model)
assert isinstance(result["answer"], str) and result["answer"].strip()
assert isinstance(result["sources"], list)
def test_cites_only_retrieved_documents():
# A source the retriever never returned is a hallucinated citation.
result = answer("How long does shipping take?", fake_model)
assert set(result["sources"]) <= set(result["retrieved"])
Why the fake model? It tests your code: prompt building, parsing, retrieval wiring. It doesn’t evaluate the model, and that’s deliberate. If a contract test fails, you know the bug is yours.
Layer 2: Golden dataset
A golden dataset is a curated list of questions with the facts the answer must include and the phrases it must not include. It’s the backbone of LLM evaluation and the closest thing it has to a regression suite. If your prompts change often, pair it with a prompt regression suite.
# ── Layer 2: golden dataset (property checks, not exact strings) ────────
GOLDEN = [
# (question, facts that must appear, phrases that must not appear)
("Can I get a refund?", ["30 days"], ["60 days", "no refunds"]),
("How long is shipping?", ["5 to 7"], ["overnight"]),
("Do you sell gift cards?", ["don't know"], ["yes", "gift cards are"]), # out-of-scope
]
@pytest.mark.parametrize("question,must_have,must_not", GOLDEN)
def test_golden_answers(question, must_have, must_not):
text = answer(question, fake_model)["answer"].lower()
for fact in must_have:
assert fact.lower() in text, f"missing fact: {fact!r}"
for bad in must_not:
assert bad.lower() not in text, f"forbidden phrase: {bad!r}"
Notice the third row. The out-of-scope question is the most valuable case in the set. A support bot that confidently invents a gift-card policy is worse than one that says “I don’t know.” Most teams forget to evaluate refusals.
Trade-off: substring checks are cheap and transparent, but brittle with synonyms (“a month” vs “30 days”). Start here anyway. Move a case to the judge only when substring checks give false failures on that case.
Layer 3: Pass-rate thresholds
This is how you handle non-determinism honestly. Run the same case many times and gate on the pass rate.
# ── Layer 3: pass-rate threshold for non-deterministic output ───────────
def flaky_model(prompt: str, rng=random.Random(7)) -> str:
"""Simulates a real model: right ~90% of the time. Seeded so CI is repeatable."""
if rng.random() < 0.1:
return json.dumps({"answer": "Refunds are possible within 60 days.",
"sources": ["refund-policy"]})
return fake_model(prompt)
def test_refund_fact_pass_rate():
runs = 50
passed = sum("30 days" in answer("Can I get a refund?", flaky_model)["answer"]
for _ in range(runs))
# One sample proves nothing. Gate on a rate you agreed with the product owner.
assert passed / runs >= 0.8, f"pass rate {passed}/{runs}"
Why a threshold instead of retries? A @retry(3) decorator turns a 70%-reliable feature into a green build and hides the risk from everyone. A threshold makes the reliability number explicit, and someone has to sign off on it.
Pick the threshold by risk, not by what currently passes. A refund amount shown to customers might need 99%. A marketing tagline generator might be fine at 70%.
Layer 4: LLM-as-a-judge
Some qualities can’t be checked with code: “Is every claim supported by the context?” or “Is the tone appropriate?” For these, an LLM-as-a-judge (a second model) scores the output against a rubric.
# ── Layer 4: LLM-as-judge for qualities you can't regex ─────────────────
JUDGE_RUBRIC = """Score the ANSWER against the CONTEXT from 1 to 5.
5 = every claim is supported by CONTEXT. 1 = contains a claim CONTEXT contradicts.
Reply as JSON: {"score": int, "reason": str}"""
def fake_judge(prompt: str) -> str:
"""Replace with a call to a different (ideally stronger) model than the one under test."""
contradicted = "60 days" in prompt and "30 days" in prompt
return json.dumps({"score": 1 if contradicted else 5,
"reason": "contradicts context" if contradicted else "grounded"})
def judge_faithfulness(answer_text: str, context: str, judge=fake_judge) -> int:
verdict = json.loads(judge(f"{JUDGE_RUBRIC}\n\nCONTEXT: {context}\n\nANSWER: {answer_text}"))
return int(verdict["score"])
def test_judge_flags_contradiction():
# Test the judge itself on known-bad input before you trust its scores.
ctx = "Refunds are available within 30 days of purchase."
assert judge_faithfulness("Refunds within 60 days.", ctx) == 1
assert judge_faithfulness("Refunds within 30 days.", ctx) == 5
The last test is the one most teams skip. A judge is itself an untested LLM. Feeding it known-bad answers is the smoke test. The next section is the real calibration.
Three judge design choices that matter: – Use a narrow rubric with anchored scores. “Rate quality 1–10” gives noise. “5 = every claim supported, 1 = a contradiction” gives signal. – Ask for the reason too. It makes failures debuggable and exposes a judge that’s guessing. – Use one criterion per judge call. A judge asked to score faithfulness, tone and completeness at once blends them into one vague number.
How do you calibrate an LLM-as-a-judge?
Calibration answers one question: when the judge says “pass,” would a careful human agree? You need two things: a labelled sample and the right metric.
Step 1: label a sample. Take 50–100 real outputs and have people score them with the same rubric the judge uses. Include deliberately bad answers. If every answer in the sample is good, you can’t learn anything about the judge.
Step 2: compare with kappa, not raw agreement. Raw agreement flatters a lazy judge. If 90% of answers are good, a judge that passes everything agrees with humans 90% of the time. Cohen’s kappa discounts the agreement you’d get by chance and exposes it.
# test_judge_calibration.py — run with: pytest -q
# Calibrate the judge before you trust it: compare it with humans, then probe for bias.
def judge_vs_humans(judge_scores, human_scores, pass_mark=4):
"""Compare pass/fail verdicts. Returns agreement, Cohen's kappa and the two error types."""
pairs = [(j >= pass_mark, h >= pass_mark) for j, h in zip(judge_scores, human_scores)]
n = len(pairs)
agreement = sum(j == h for j, h in pairs) / n
# Kappa discounts agreement you'd get by chance. If 90% of answers pass,
# a judge that always says "pass" scores 90% agreement and kappa 0.
judge_pass = sum(j for j, _ in pairs) / n
human_pass = sum(h for _, h in pairs) / n
chance = judge_pass * human_pass + (1 - judge_pass) * (1 - human_pass)
kappa = 1.0 if chance == 1 else (agreement - chance) / (1 - chance)
false_pass = sum(j and not h for j, h in pairs) # judge passed what humans failed: the risky one
false_fail = sum(h and not j for j, h in pairs) # judge failed what humans passed: noise
return {"agreement": agreement, "kappa": round(kappa, 2),
"false_pass": false_pass, "false_fail": false_fail}
def test_lazy_judge_looks_good_on_agreement_but_not_kappa():
humans = [5, 5, 5, 5, 5, 5, 5, 5, 5, 1] # 9 good answers, 1 bad
lazy = [5] * 10 # a judge that passes everything
report = judge_vs_humans(lazy, humans)
assert report["agreement"] == 0.9 # looks great...
assert report["kappa"] == 0.0 # ...but it has learned nothing
assert report["false_pass"] == 1 # and it let the bad answer through
def test_useful_judge_scores_well_on_both():
humans = [5, 4, 5, 1, 2, 5, 4, 1, 5, 2]
judge = [5, 5, 4, 1, 1, 5, 4, 2, 3, 2] # one disagreement (a false fail)
report = judge_vs_humans(judge, humans)
assert report["agreement"] == 0.9
assert report["kappa"] >= 0.6
assert report["false_pass"] == 0
Both judges score 90% agreement. Only kappa tells them apart: 0.0 for the lazy judge, 0.8 for the useful one.
Step 3: weigh the two errors differently. A false pass (the judge approves what humans reject) ships bad answers. A false fail only costs a human a second look. For high-risk features, tune the rubric and pass mark until false passes are near zero, even if false fails rise.
Step 4: probe for known biases. Research on LLM judges, notably the MT-Bench study by Zheng et al. (2023), documents three biases worth testing for:
- Position bias: in pairwise comparisons, the judge favours whichever answer appears first (or second).
- Verbosity bias: the judge favours longer answers even when they add nothing.
- Self-preference: the judge favours output from its own model family.
Position bias is the easiest to test, because a fair judge’s verdict shouldn’t change when you swap the order:
# ── Position bias: a pairwise judge must not care which answer comes first ──
def consistent_under_swap(judge, question, a, b) -> bool:
"""Ask twice with the order swapped. A fair judge picks the same answer both times."""
first = judge(question, a, b) # returns "A" or "B"
swapped = judge(question, b, a)
return (first == "A") == (swapped == "B")
def biased_judge(question, a, b):
return "A" # always prefers whatever it reads first
def fair_judge(question, a, b):
return "A" if "30 days" in a else "B" # prefers the correct answer wherever it sits
def test_swap_check_catches_position_bias():
q, good, bad = "Refund window?", "Within 30 days.", "Within 60 days."
assert not consistent_under_swap(biased_judge, q, good, bad)
assert consistent_under_swap(fair_judge, q, good, bad)
With a real judge, run the swap check across your whole pairwise set and report the consistency rate. For verbosity bias, pad a correct answer with filler and check that its score doesn’t rise. For self-preference, use a judge from a different model family than the model under test.
Re-calibrate whenever the judge model, its prompt or the rubric changes. A judge upgrade is a change to your test oracle, and it deserves the same scrutiny as a change to the product.
How do you evaluate a RAG application for hallucinations?
RAG (retrieval-augmented generation) fails in two separate places, and you should evaluate them separately:
- Retrieval: did the right documents come back? Test this without the LLM at all, by asserting the expected
doc_idappears in the retrieved set for each golden question. It’s fast, deterministic and cheap. - Generation: given the right documents, did the model stay faithful to them? Use the source-subset check (Layer 1), forbidden facts (Layer 2) and a faithfulness judge (Layer 4).
If you only evaluate end to end, a wrong answer doesn’t tell you whether to fix the index or the prompt. Splitting the layers turns one vague failure into a pointer to the broken component. For retrieval metrics in depth, see our RAG testing framework guide.
Where does LLM evaluation run in CI?
| Layer | Model used | When it runs | Blocks merge? |
|---|---|---|---|
| Contract checks | Fake | Every commit | Yes |
| Golden dataset (fake) | Fake | Every commit | Yes |
| Golden dataset + pass-rate (real) | Real model | Nightly, and on prompt/model changes | Yes for prompt changes |
| LLM-as-a-judge | Real judge model | Nightly / pre-release | Report-only until calibrated |
| Judge calibration | Real judge + human labels | On judge, rubric or prompt change | Gates whether the judge may block |
Why split it this way? Real-model evals cost money, take minutes and are statistically noisy. Running them on every commit trains developers to ignore red builds. Run them where a change can actually affect model behaviour, such as a prompt edit, a model upgrade or a knowledge-base update.
Why does the judge start as report-only? An uncalibrated judge that blocks merges is a random number generator with veto power. Let it block only after its kappa against human labels meets the bar you set.
Track eval scores over time, not just pass/fail. A slide from 96% to 88% across three model updates is a story no single build will tell you.
Practical examples: beginner to advanced
Beginner: a JSON contract. Your app asks the model for {"category": ...} from a fixed list. Assert it parses, and that category is in the allowed set. This alone catches a surprising share of production incidents.
Intermediate: a refusal suite. Write 20 questions your bot must not answer: legal advice, competitor pricing, other customers’ data. Assert a refusal phrase or an empty sources list. Run it against the real model nightly with a 95% threshold.
Advanced: judge calibration as a release gate. Keep 100 human-labelled outputs in version control. When anyone changes the judge model or rubric, CI re-scores them and fails if kappa drops below your bar or any false pass appears on the high-risk subset.
Common mistakes testers make
| Mistake | Why it happens | Better approach |
|---|---|---|
| Asserting exact output strings | Habit from deterministic testing | Check facts, structure and forbidden content |
| Running each case once | Treating LLMs like functions | Repeat runs and gate on a pass rate |
| Adding retries to make tests green | Pressure to unblock the pipeline | Make the reliability threshold explicit and owned |
| Trusting the judge blindly | It “sounds” authoritative | Calibrate against human labels before it can block |
| Reporting raw agreement for the judge | It’s the obvious number | Report kappa plus false-pass count |
| Pairwise judging in one order only | Half the API calls | Swap order and require a consistent verdict |
| One judge call scoring many criteria | Fewer calls | One criterion per call |
| Only evaluating happy paths | Demos drive the dataset | Add out-of-scope, adversarial and refusal cases |
Interview questions
Beginner 1. What is LLM evaluation? Measuring output quality against defined criteria across a dataset, instead of asserting exact strings. 2. What is a golden dataset? A curated set of inputs with required and forbidden facts that acts as your regression suite. 3. What is LLM-as-a-judge? Using a second LLM with a rubric to score outputs on qualities code can’t check. 4. Why use a fake model in unit tests? It isolates your code’s bugs from model behaviour, and keeps tests fast, free and deterministic. 5. What does temperature do for evaluation? It reduces randomness but doesn’t guarantee identical output, so you still need property checks.
Intermediate 1. How do you handle non-determinism in CI? Run cases N times and gate on a pass rate agreed with the product owner. 2. What makes a good judge rubric? One criterion, anchored score levels and a required reason. 3. Why test retrieval separately in RAG? So a failure points to the index or the prompt, not just “wrong answer.” 4. What is position bias? A pairwise judge favouring an answer because of where it appears, not what it says. 5. When should real-model evals run? On prompt, model or knowledge-base changes, plus nightly, not on every commit.
Advanced 1. Why use Cohen’s kappa instead of agreement to calibrate a judge? Agreement is inflated when one verdict dominates. Kappa removes chance agreement. 2. Which judge error is worse, a false pass or a false fail? A false pass, because it ships bad output. A false fail only costs review time. 3. How do you detect verbosity bias? Pad a correct answer with filler and check that its score doesn’t increase. 4. When must you re-calibrate a judge? Whenever the judge model, judge prompt or rubric changes. 5. How do you keep eval costs under control? Tier the suites, cache unchanged cases, and reserve the judge for checks code can’t do.
How AI changes LLM evaluation itself
Tools like ChatGPT, Claude, Gemini and Copilot can now generate golden-dataset cases, adversarial prompts and judge rubrics in minutes. That’s genuinely useful for breadth.
The catch is that a model generating its own evaluation cases tends to generate cases it already handles well. Use AI to draft the dataset, then have a human add the cases that come from real incidents, support tickets and domain knowledge. Those are the ones that find bugs.
Judges will keep improving, but the principle won’t change: an oracle you haven’t tested is an opinion. The tester’s job shifts from scoring every answer to owning the labelled sample that keeps the judge honest.
Evaluation is also moving closer to observability: production traces sampled and scored continuously, with failures promoted into the golden set. If your LLM calls tools, the same thinking extends to testing AI agents.
Best practices checklist
- The model is injected so unit tests use a fake
- Contract checks run on every commit
- The golden dataset includes out-of-scope and refusal cases
- Checks target facts and forbidden content, not exact strings
- Real-model cases use pass-rate thresholds with named owners
- Each judge call scores one criterion with an anchored rubric
- The judge is calibrated against human labels using kappa
- False passes are tracked separately from false fails
- Pairwise judges pass an order-swap consistency check
- The judge re-calibrates when its model, prompt or rubric changes
- RAG retrieval is evaluated separately from generation
- Production failures get added to the golden set
Tool comparison
You don’t need a framework to start. The pytest code above is a complete evaluation harness. Frameworks earn their place once you need dashboards, dataset versioning or shared judge metrics.
| Need | Plain pytest | promptfoo | DeepEval | Ragas |
|---|---|---|---|---|
| Language | Python | Config-driven (YAML) + JS/TS | Python (pytest-style) | Python |
| Best for | Full control, small suites | Comparing prompts/models side by side | Ready-made metrics in a test runner | RAG-specific metrics |
| Built-in LLM judge metrics | Write your own | Yes | Yes | Yes |
| Judge calibration | Write your own (above) | Manual | Manual | Manual |
| Learning curve | Lowest | Low | Medium | Medium |
| Lock-in | None | Low | Medium | Medium |
Whichever tool you pick, calibrate its built-in judge metrics against your own labelled sample. A metric named “faithfulness” is only as good as its agreement with your humans. Check each tool’s current docs before adopting it, since these projects ship changes quickly.
Conclusion
LLM evaluation isn’t new magic. It’s old testing discipline applied honestly: define what “correct” means, check it cheaply wherever you can, measure it statistically where you can’t, and never trust an oracle you haven’t tested.
That last part is where most teams fail. An LLM-as-a-judge is a test oracle like any other, and it needs human-labelled evidence before it gets to block a release.
Start small. Inject your model, write five contract checks and a ten-row golden set with two out-of-scope questions, then label 50 outputs and measure your judge’s kappa. You’ll learn more about your system in a week than a month of demos taught you.
Tools change. Testing principles endure.
Keep testing. Keep learning. Keep questioning.
Go deeper · QA Bash kit
LLM Evals Kit
Test LLM features like any other feature: a dataset, graders, an LLM judge and a CI gate.
3–6 yrs · 32 passing tests
₹999₹1,249
Rate this article
8.3/10 average · 41 ratings
Discussion
Be the first to weigh in
Your question or experience helps the next tester who reads this. Sign in and earn +25 XP for every approved comment or reply.
Written by
Frequently asked questions.
Keep reading




