Lesson 1 - Why and When to Use an LLM Judge

Welcome to Why and When to Use an LLM Judge#

In Module 3 you scored Docent – our assistant for the Meridian serverless database docs – with exact match, token F1, and ROUGE/BLEU. Every one of those metrics compares strings, and every one of them broke the same way on the same kind of answer: a fluent, confidently wrong answer that recycled the reference’s words scored higher than a correct one. The ROUGE example was the clearest warning – an answer that said the literal opposite of the truth, by inserting the word “not,” topped the leaderboard because it reused almost every word of the reference. A formula that only counts shared tokens has no way to know that one inserted “not” flipped true into false, or that “twenty-five bucks a month” and “25 dollars per month” mean the same thing.

That gap is where this module lives. When an answer is open-ended, paraphrased, or judgement-heavy, correctness depends on meaning, not surface overlap – and meaning is exactly what a capable language model can assess. The idea is disarmingly simple: instead of scoring the answer with arithmetic, you hand it to a model and ask it to grade. That is the LLM-as-judge pattern, and by the end of this lesson you’ll have a working reference-based judge that gets right the exact Docent cases token F1 got wrong.

In this lesson, you will:

• See precisely why a deterministic metric cannot separate a correct paraphrase from a fluent wrong answer – with the real F1 numbers from Module 3
• Learn what an LLM judge is: prompt a model with the question, the answer, and (optionally) a reference or rubric, and ask for a verdict plus reasoning
• Preview the module’s three judging modes – reference-based, reference-free rubric scoring, and pairwise comparison – and build the simplest one live
• Weigh the judge’s strengths (meaning, paraphrase, partial credit, scale) against its risks (bias, non-determinism, gaming), and know when a plain deterministic check is the better tool

The problem a formula cannot solve#

Recall the mechanism from Module 3: string metrics reward surface overlap, not truth. That produces two opposite errors on question answering. A correct answer phrased in fresh words shares few tokens with the reference and gets a low score (a false negative), while a wrong answer that reuses the reference’s vocabulary shares many tokens and gets a high score (a false positive – the dangerous one, because your eval cheers for an answer that would mislead a user).

Let’s make that concrete with real Docent answers and the same token-F1 definition you built in Module 3 (normalize, then F1 over the token multiset). We score four answers whose truth we already know – two correct paraphrases and two fluent wrong answers – and watch F1 do exactly the wrong thing. This block is pure arithmetic over fixed strings, so its numbers are identical on every run:

import re, string
from collections import Counter

def normalize(text):
    text = text.lower()
    text = "".join(ch for ch in text if ch not in set(string.punctuation))
    text = re.sub(r"\b(a|an|the)\b", " ", text)     # drop articles
    return " ".join(text.split())                    # collapse whitespace

def exact_match(pred, ref):
    return int(normalize(pred) == normalize(ref))

def f1_score(pred, ref):
    pred_toks = normalize(pred).split()
    ref_toks = normalize(ref).split()
    if not pred_toks or not ref_toks:
        return 0.0
    shared = sum((Counter(pred_toks) & Counter(ref_toks)).values())
    if shared == 0:
        return 0.0
    precision = shared / len(pred_toks)
    recall = shared / len(ref_toks)
    return 2 * precision * recall / (precision + recall)

ITEMS = [
    dict(truth="correct",
         question="Can you change a database's region after it is created?",
         reference="No; the region is fixed at creation and you must export and "
                   "re-import into a new database to move regions.",
         answer="A database's location is permanent once it is set up, so relocating "
                "it means dumping the data and loading it into a fresh instance elsewhere."),
    dict(truth="incorrect",
         question="Can you change a database's region after it is created?",
         reference="No; the region is fixed at creation and you must export and "
                   "re-import into a new database to move regions.",
         answer="Yes, the region is not fixed at creation and you do not need to "
                "export and re-import into a new database to move regions."),
    dict(truth="correct",
         question="How much does the Pro plan cost per month?",
         reference="25 dollars per month",
         answer="The Pro tier will run you twenty-five bucks every thirty days."),
    dict(truth="incorrect",
         question="How much does the Pro plan cost per month?",
         reference="25 dollars per month",
         answer="Pro costs 250 dollars per month."),
]

print(f"{'truth':<12}{'EM':>4}{'F1':>7}   answer")
print("-" * 74)
for it in ITEMS:
    em = exact_match(it["answer"], it["reference"])
    f1 = f1_score(it["answer"], it["reference"])
    it["em"], it["f1"] = em, f1
    print(f"{it['truth']:<12}{em:>4}{f1:>7.2f}   {it['answer'][:44]}...")

truth        EM     F1   answer
--------------------------------------------------------------------------
correct        0   0.15   A database's location is permanent once it i...
incorrect      0   0.80   Yes, the region is not fixed at creation and...
correct        0   0.00   The Pro tier will run you twenty-five bucks ...
incorrect      0   0.60   Pro costs 250 dollars per month....

Read the F1 column against the truth column and the failure is stark. The two correct paraphrases score 0.15 and 0.00 – near or at the floor – because they say the right thing in different words. The two incorrect answers score 0.80 and 0.60 – far higher than the correct ones – because “the region is not fixed” reuses the reference’s words while flipping the meaning, and “250 dollars per month” shares dollars per month with the reference while being off by a factor of ten. Exact match is no help either: it’s 0 for all four, so it can’t even separate the correct ones from a total miss. If you ranked these four answers by F1, you would rank both wrong answers above both right ones. No threshold on F1 fixes this, because the wrong answers genuinely share more tokens than the right ones. The signal a formula needs simply isn’t in the token counts.

What an LLM judge is#

An LLM judge is a language model prompted to grade another model’s output. You give the judge the material a human grader would need – the question, the answer under test, and usually a reference answer and/or a rubric – and you ask it to return a verdict (and, importantly, a short reason). Because the judge is a capable model reading the answer for meaning, it can do what a formula can’t: recognize that “twenty-five bucks every thirty days” and “25 dollars per month” assert the same fact, and that “the region is not fixed” asserts the opposite of the reference.

There are three main ways to pose the grading task, and this module covers all three:

• Reference-based grading – “Here is a known-correct reference; is this answer consistent with it?” The judge outputs correct/incorrect (or a graded score). This is the mode you’ll build in this lesson, and it maps directly onto Docent’s golden Q&A set, where every question has a trusted reference answer.
• Reference-free direct scoring – there is no single right answer, so you give the judge a rubric (e.g. “score 1–5 on faithfulness to the docs and helpfulness”) and it scores the answer against those criteria. This is Lesson 2.
• Pairwise comparison – “Here are two answers to the same question; which is better?” The judge picks A or B (or a tie). This is Lesson 3, and it’s the backbone of preference evaluation and leaderboards.

The unit of work is always the same: a carefully written prompt that frames the grading task, plus a request for structured output (so you can parse the verdict programmatically instead of scraping prose). We’ll ask for a tiny JSON object – {"verdict": ..., "reason": ...} – which is enough to score at scale while keeping a human-readable rationale for every decision.

Building a reference-based judge for Docent#

Now the payoff. We hand the same four items that fooled F1 to claude-haiku-4-5 and ask it to grade each answer against its reference as correct or incorrect, returning a small JSON object. The judge prompt states the rule plainly – convey the same facts as the reference; a paraphrase counts as correct; contradicting the reference is incorrect – and asks for JSON only, so the verdict is easy to parse. Note the client reads the API key from the environment (ANTHROPIC_API_KEY); no key ever appears in the code.

import json
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

JUDGE_PROMPT = """You are grading a documentation assistant's answer against a known-correct reference.
Decide whether the answer is CORRECT: it must convey the same facts as the reference.
A correct paraphrase counts as correct. An answer that states something the reference
contradicts is incorrect.

Question: {question}
Reference answer: {reference}
Assistant answer: {answer}

Reply with ONLY a JSON object: {{"verdict": "correct" or "incorrect", "reason": "<one short sentence>"}}"""

def judge(question, reference, answer, model="claude-haiku-4-5"):
    prompt = JUDGE_PROMPT.format(question=question, reference=reference, answer=answer)
    msg = client.messages.create(model=model, max_tokens=150,
                                 messages=[{"role": "user", "content": prompt}])
    text = msg.content[0].text.strip()
    text = text[text.find("{"): text.rfind("}") + 1]   # keep just the JSON object
    return json.loads(text)

print(f"{'truth':<11}{'F1':>6}   judge      match?")
print("-" * 74)
for it in ITEMS:                      # ITEMS + f1 values from the block above
    v = judge(it["question"], it["reference"], it["answer"])
    match = "OK" if v["verdict"] == it["truth"] else "MISS"
    print(f"{it['truth']:<11}{it['f1']:>6.2f}   {v['verdict']:<9}  [{match}]")
    print(f"    reason: {v['reason']}")

truth          F1   judge      match?
--------------------------------------------------------------------------
correct      0.15   correct    [OK]
    reason: The assistant's answer accurately conveys that the region cannot be changed
    after creation and that data must be exported and re-imported to move to a different
    region, using equivalent terminology.
incorrect    0.80   incorrect  [OK]
    reason: The assistant's answer directly contradicts the reference by claiming the region
    can be changed and that export/re-import is not necessary, when the reference explicitly
    states the opposite.
correct      0.00   correct    [OK]
    reason: The assistant's answer conveys the same cost ($25 per month) using different
    wording ('twenty-five bucks' and 'every thirty days' are equivalent to the reference's
    '$25' and 'per month').
incorrect    0.60   incorrect  [OK]
    reason: The assistant states the Pro plan costs $250 per month, which contradicts the
    reference answer of $25 per month.

The judge got all four right, and it got right precisely the cases F1 got wrong. The correct paraphrase that F1 scored 0.15 – near the floor – the judge marks correct, and it explains why: same facts, different terminology. The fluent wrong answer that F1 scored 0.80 – near the ceiling – the judge marks incorrect, and it names the exact fault: it contradicts the reference. Same for the pricing pair: the twenty-five bucks paraphrase (F1 0.00) is graded correct and the 250 dollars answer (F1 0.60) is graded incorrect. Where the formula ranked both wrong answers above both right ones, the judge separated them cleanly – and left an audit trail of one-sentence reasons you can read and trust or contest.

Strengths, risks, and when NOT to use a judge#

The demo shows the strengths at a glance. A judge handles paraphrase and meaning (it credits “twenty-five bucks” as “25 dollars”), it can give partial credit and grade tone or style against a rubric, and it can assess things no formula touches – was the answer helpful, was it faithful to the docs, did it appropriately refuse an out-of-scope question. And it does all this at a fraction of the cost and latency of a human grader, so you can run it over thousands of Docent answers on every change instead of sampling a handful for a human review.

But a judge is itself a model, and that carries risks you must respect – we preview them here and tackle them head-on in Lesson 4:

• It can be biased or simply wrong. Judges tend to favor longer answers, the first option shown in a pairwise test, and outputs written in their own style – systematic biases, not random noise.
• It is non-deterministic. The same answer can get different verdicts on different runs, so a single judgement is a sample, not ground truth.
• It can be gamed. An answer optimized to look authoritative can talk a naive judge into a high score. A judge you never checked against human labels is a judge you can’t trust – which is why calibration (measuring judge–human agreement) is non-negotiable before you rely on one.

Which leads to the most important discipline in this whole module: don’t use a judge when a deterministic check will do. If the correct answer is an exact value – an HTTP status code (429), an environment variable name (MERIDIAN_API_KEY), a plan price – a normalized exact-match or a targeted “does the answer contain 429?” check is cheaper, faster, perfectly reproducible, and not gameable. Same for structure: if Docent must return valid JSON with an answer and a doc_id, a schema check answers “is this the right shape?” with total certainty, and no model call can beat that. Spend the judge on what only a judge can do – meaning, quality, and open-ended correctness – and let deterministic checks handle everything with a crisp right answer. The right eval suite is a layered one: cheap exact checks first, a judge only where they run out.

Practice Exercises#

Exercise 1: Find an item where F1 and the judge disagree#

Extend ITEMS with a new correct paraphrase and a new fluent wrong answer drawn from the Meridian docs – for example, the backups question (reference "90 days" for the Scale plan) with a correct paraphrase like "Scale keeps backups for three months." and a wrong-but-overlapping "Backups are retained for 9 days on Scale.". Run both the F1 block and the judge block on your new items and confirm the judge separates correct from incorrect where F1 does not.

Exercise 2: Grade an out-of-scope refusal#

Docent’s golden set includes an out-of-scope probe: "Does Meridian support MongoDB-style aggregation pipelines?", where the correct behavior is to refuse (“I don’t have that in the docs.”). A reference-based judge needs to handle “no reference answer exists.” Adapt the judge prompt so that when the reference says the docs don’t cover the question, a Docent answer that refuses is graded correct and a Docent answer that makes something up is graded incorrect. Test it on a good refusal and a confident fabrication.

Exercise 3: Decide judge vs. deterministic check for five questions#

For each of these five Docent questions, decide whether you’d grade it with a deterministic check or an LLM judge, and justify it in one line: (a) “What HTTP status code is returned when you exceed the rate limit?” (b) “Explain in your own words why you can’t change a region after creation.” (c) “Which environment variable does the Python SDK read?” (d) “Is Docent’s tone appropriate for a nervous first-time user?” (e) “Return the answer and its source doc id as JSON.” Then, for the ones you assigned to a deterministic check, sketch the def check(...) you’d write.

Summary#

Deterministic metrics compare strings, so they cannot tell a correct paraphrase from a fluent wrong answer – exactly the failure Module 3 exposed, where an answer meaning the opposite of the truth out-scored a correct one. You reproduced it on four real Docent items: two correct paraphrases scored token F1 of 0.15 and 0.00, while two fluent wrong answers scored 0.80 and 0.60, so ranking by F1 would put both wrong answers above both right ones. An LLM judge closes that gap by grading for meaning: prompt a capable model (claude-haiku-4-5) with the question, the answer, and a reference, and ask for a structured verdict with a reason. Run live on those same four items, the judge marked both paraphrases correct and both fluent wrong answers incorrect – getting right precisely the cases F1 got wrong, with a one-sentence rationale for each. The module explores three judging modes (reference-based, reference-free rubric scoring, and pairwise); this lesson built the simplest, reference-based correct/incorrect grading. A judge handles paraphrase, meaning, partial credit, and tone, and scales far cheaper than humans – but it can be biased, is non-deterministic, and can be gamed, so it must be calibrated. And when a fixed rule can verify the answer (an exact ID, valid JSON, a number), that deterministic check is cheaper, exact, and the right tool. Use a judge only where a formula runs out.

Key Concepts#

• LLM-as-judge – prompting a capable model to grade another model’s output, given the question, the answer, and usually a reference and/or rubric, returning a verdict plus reasoning.
• Why formulas fail on QA – string metrics reward surface overlap, so a correct paraphrase (few shared tokens) scores low and a fluent wrong answer (many shared tokens) scores high; the four-item demo shows F1 ranking both wrong answers above both right ones.
• Reference-based grading – the mode built here: judge an answer as correct/incorrect against a known-correct reference; ideal for Docent’s golden Q&A set.
• The three judging modes – reference-based (vs. a reference), reference-free direct scoring (vs. a rubric, Lesson 2), and pairwise comparison (A vs. B, Lesson 3).
• Structured verdicts – ask the judge for small JSON like {"verdict": ..., "reason": ...} so results parse programmatically while keeping a human-readable rationale.
• Judge risks – bias (length, position, self-preference), non-determinism (verdicts vary run to run), and gamability, which is why calibration against human labels (Lesson 4) is mandatory.
• When NOT to use a judge – if a fixed rule can verify the answer (exact IDs, valid JSON, number match), a deterministic check is cheaper, exact, reproducible, and not gameable.

Why This Matters#

Most teams meet the LLM-as-judge pattern the moment their eval suite plateaus: the deterministic metrics all pass, yet users still hit wrong answers, because the metrics were quietly rewarding fluent nonsense that reused the right words. Knowing when a judge is the right instrument – and, just as important, when a two-line deterministic check beats it on cost, speed, and certainty – is what separates an eval suite you can trust from one that gives you false confidence. The demo in this lesson is the whole argument in miniature: the same four answers that F1 ranked backwards, a judge sorted correctly, because it read them for meaning instead of counting tokens. That capability is powerful and, used carelessly, dangerous – a biased, un-calibrated judge produces confident wrong grades at scale. The rest of this module turns the raw pattern into a disciplined tool: writing rubrics, running pairwise comparisons, and calibrating the judge against humans so its scores are ones you have earned the right to believe.

Continue Building Your Skills#

You now know why a formula can’t grade meaning, what an LLM judge is, and how to build the simplest one – reference-based correct/incorrect grading – and you watched it fix the exact Docent cases where token F1 was misled. You also know the discipline that keeps a judge honest: use it only where a deterministic check can’t, and remember that its verdicts are samples from a model that can be biased and gamed. Next, in Lesson 2, you’ll drop the reference and grade with a rubric instead: when there’s no single right answer, you’ll define what “good” means – faithful to the docs, helpful, appropriately scoped – and have claude-haiku-4-5 return a numeric score against those criteria. That’s the mode you reach for when the question isn’t “does this match the reference?” but “how good is this answer, really?”