Lesson 5 - Guided Project: A Calibrated Judge for Docent

Welcome to the Guided Project#

This is the capstone for LLM-as-Judge, and it pulls the whole module into one artifact you can run: a calibrated judge for Docent. Over the last four lessons you learned when a model judge is the right instrument, how to write a rubric and get structured direct scores with rationales, how to run pairwise comparisons, and how position, verbosity, and self-preference bias a naive judge. This project welds those pieces into a single trustworthy scorer — and, crucially, refuses to report a number until it has earned it.

The four stages mirror how a careful team stands up a judge in production. You define the rubric and the judge function — a judge_answer call on claude-haiku-4-5 that returns validated JSON with its reasoning before its verdict; you run the judge over Docent’s golden set live and collect verdicts; you calibrate those verdicts against human gold labels you author in code, measuring agreement and Cohen’s kappa against a threshold you commit to in advance; and you report Docent’s quality score with its calibration caveat attached. The judge itself is a stochastic model, so its verdicts can wobble run to run — but the calibration math on fixed label arrays is deterministic, and the whole discipline is refusing to say “Docent is X% correct” without immediately saying “by a judge that agrees with humans Y% of the time.”

By the end of this project, you will be able to:

• Write a judge_answer(question, answer, reference) function on claude-haiku-4-5 that returns validated JSON with a rationale before a correct/incorrect verdict
• Run the judge live over Docent’s golden set — including a paraphrase and the out-of-scope refusal — and collect real verdicts
• Calibrate the judge against hand-authored human gold labels, computing agreement rate and Cohen’s kappa, and decide trustworthiness against a stated threshold
• Report a judged quality score for Docent with its calibration caveat attached, and inspect the disagreements instead of averaging them away

Stage 1: Define the Rubric and the Judge Function#

A judge is only as good as the rubric it applies, so we write the rubric down first — in the system prompt, in plain language: a candidate is correct if it matches the reference in fact, even if it is phrased differently or adds helpful detail, and incorrect if it contradicts the reference or is vaguer than the reference requires. The out-of-scope rule matters too: when the reference says the answer isn’t in the docs, a candidate that refuses is correct and one that invents an answer is incorrect.

Two design choices from earlier in the module are baked in. First, reasoning before verdict — the JSON’s reasoning field comes before verdict, so the model commits its thinking before it commits its vote, which is worth a real accuracy bump for a judge. Second, we reuse Module 3’s structured-output discipline: the judge must return JSON, we extract and json.loads it, and we validate that the verdict is one of two allowed strings and the rationale is non-empty — a malformed judgement raises rather than silently scoring.

import warnings; warnings.filterwarnings("ignore")
import json, re
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

def parse_json(text):
    m = re.search(r"\{.*\}", text, re.DOTALL)   # first {...} block, forgiving of prose around it
    if not m:
        raise ValueError("no JSON object found in judge output")
    return json.loads(m.group(0))

# --- the rubric lives in the system prompt, in plain language ---
JUDGE_SYSTEM = (
    "You are a strict grader for a documentation assistant. You receive a QUESTION, a "
    "REFERENCE answer known to be correct, and a CANDIDATE answer to grade. Decide whether "
    "the candidate is factually correct with respect to the reference. A correct candidate may "
    "be phrased differently or add helpful detail, as long as it does not contradict the "
    "reference and is not vaguer than the reference requires. If the reference says the answer "
    "is not in the docs, a candidate that refuses or says it doesn't know is CORRECT, and a "
    "candidate that invents an answer is INCORRECT. First think, then decide: reply with ONLY a "
    'JSON object of the form {"reasoning": "<one or two sentences>", "verdict": "correct" or '
    '"incorrect"}, with the reasoning field BEFORE the verdict field.'
)

def judge_answer(question, answer, reference, model="claude-haiku-4-5"):
    user = (f"QUESTION: {question}\n\nREFERENCE (known correct): {reference}\n\n"
            f"CANDIDATE (to grade): {answer}\n\nGrade the candidate.")
    msg = client.messages.create(model=model, max_tokens=250, system=JUDGE_SYSTEM,
                                 messages=[{"role": "user", "content": user}])
    data = parse_json(msg.content[0].text)
    # validate: enforce the contract, don't trust the model's shape
    if data.get("verdict") not in ("correct", "incorrect"):
        raise ValueError(f"invalid verdict: {data.get('verdict')!r}")
    if not isinstance(data.get("reasoning"), str) or not data["reasoning"].strip():
        raise ValueError("missing or empty reasoning")
    return data

# two fixed sanity cases: an obviously-right and an obviously-wrong candidate
print("case A:", judge_answer("How much does the Pro plan cost per month?",
                              "The Pro plan costs 25 dollars per month.", "25 dollars per month"))
print("case B:", judge_answer("How much does the Pro plan cost per month?",
                              "The Pro plan costs 199 dollars per month.", "25 dollars per month"))

case A: {'reasoning': 'The candidate answer states that the Pro plan costs 25 dollars per month, which exactly matches the reference answer of 25 dollars per month. This is factually correct and uses clear, direct language.', 'verdict': 'correct'}
case B: {'reasoning': 'The candidate states the Pro plan costs $199 per month, which directly contradicts the reference answer of $25 per month. This is a factual error regarding pricing information.', 'verdict': 'incorrect'}

The judge does exactly what the rubric asks: it reasons first — noticing the 199-versus-25 contradiction in case B — and only then votes. This is a real run, so the wording of the rationale will differ slightly if you run it again, but the verdicts on cases this clear are rock-solid. The validation layer is the quiet hero here: because we assert the verdict is one of two known strings, any run where the model wanders off the JSON contract fails loudly instead of poisoning a scorecard with a silently-dropped answer.

Stage 2: Run the Judge Over Docent’s Golden Set#

Now we point the judge at Docent’s real output. We drop in the canonical Docent — keyword retrieval over the Meridian docs plus one claude-haiku-4-5 call — and the ten-item golden set, which deliberately includes a paraphrase of the pricing question (“What is the monthly price of the Pro plan?”) and the out-of-scope probe about MongoDB-style aggregation, whose only correct behavior is to refuse. For each question we generate Docent’s answer live, then judge it against the reference. Both Docent and the judge read the API key from the environment; we never hard-code one. That’s twenty small calls total — keep it modest.

DOCS = [
    {"id": "ratelimits", "title": "Rate Limits",
     "text": "The Free plan allows 60 requests per minute. The Pro plan allows 600 requests per minute. Exceeding the "
             "limit returns HTTP 429 with a Retry-After header giving the seconds to wait. Rate limits are applied per "
             "API key, not per account."},
    {"id": "plans", "title": "Plans and Pricing",
     "text": "Meridian has three plans. Free costs 0 dollars per month and includes 1 GB of storage. Pro costs 25 dollars "
             "per month and includes 50 GB of storage. Scale costs 199 dollars per month and includes 500 GB of storage. "
             "All plans include unlimited read replicas."},
    {"id": "regions", "title": "Regions",
     "text": "Databases can be created in three regions: us-east, eu-west, and ap-south. The region is fixed at creation "
             "time and cannot be changed later; to move data to another region you must export and re-import into a new "
             "database. Cross-region read replicas are available on the Scale plan only."},
    {"id": "errors", "title": "Error Codes",
     "text": "Meridian uses standard HTTP status codes. 400 means a malformed request, 401 means a missing or invalid API "
             "key, 403 means the key is valid but lacks permission, 404 means the resource does not exist, and 429 means "
             "the rate limit was exceeded. 5xx codes indicate a server-side error and should be retried with backoff."},
    {"id": "backups", "title": "Backups",
     "text": "Automatic daily backups are retained for 7 days on the Free plan, 30 days on Pro, and 90 days on Scale. "
             "Backups run at 03:00 UTC. Point-in-time recovery to any second in the retention window is available on the "
             "Scale plan. Restoring a backup creates a new database and never overwrites the original."},
    {"id": "sdks", "title": "SDKs",
     "text": "Official SDKs are available for Python, JavaScript, and Go. Install the Python SDK with 'pip install meridian'. "
             "The SDK reads the MERIDIAN_API_KEY environment variable by default. All SDKs retry idempotent requests up to "
             "3 times on 5xx errors with exponential backoff."},
    {"id": "deletion", "title": "Deleting Data",
     "text": "Deleting a database is immediate and irreversible once the 24-hour grace period ends. During the grace period "
             "the database is soft-deleted and can be restored from the dashboard. After 24 hours the data and its backups "
             "are permanently purged. Deletion requires a live-mode key with admin permission."},
]

def retrieve(question, docs, k=2):
    q = set(question.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())), reverse=True)
    return scored[:k]

def docent_answer(question, docs, model="claude-haiku-4-5"):
    ctx = "\n\n".join(f"[{d['id']}] {d['title']}\n{d['text']}" for d in retrieve(question, docs))
    prompt = (f"Answer the question using ONLY the context. If the context does not contain the answer, say "
              f"\"I don't have that in the docs.\" Be concise.\n\nContext:\n{ctx}\n\nQuestion: {question}")
    msg = client.messages.create(model=model, max_tokens=200,
                                 messages=[{"role": "user", "content": prompt}])
    return msg.content[0].text

# the golden set: 8 factual questions (incl. a paraphrase) + the out-of-scope refusal
GOLDEN = [
    dict(q="What HTTP status code does Meridian return when you exceed the rate limit?", ref="429"),
    dict(q="How many requests per minute does the Pro plan allow?", ref="600 requests per minute"),
    dict(q="What is the monthly price of the Pro plan?", ref="25 dollars per month"),   # paraphrase
    dict(q="Can you change a database's region after it is created?",
         ref="No; the region is fixed at creation and you must export and re-import into a new database to move regions."),
    dict(q="How long are backups retained on the Scale plan?", ref="90 days"),
    dict(q="Which environment variable does the Python SDK read for the API key?", ref="MERIDIAN_API_KEY"),
    dict(q="What does a 401 error mean?", ref="A missing or invalid API key"),
    dict(q="How long is the grace period before a deleted database is permanently purged?", ref="24 hours"),
    dict(q="Which plans support point-in-time recovery?", ref="Only the Scale plan"),
    dict(q="Does Meridian support MongoDB-style aggregation pipelines?",
         ref="(not in the docs -- Docent should say it doesn't know)"),   # out-of-scope refusal
]

records = []
for item in GOLDEN:
    ans = docent_answer(item["q"], DOCS)
    v = judge_answer(item["q"], ans, item["ref"])
    records.append(dict(q=item["q"], answer=ans, verdict=v["verdict"], reasoning=v["reasoning"]))
    print(f"[{v['verdict']:>9}] {item['q'][:55]}")

# show two real judge outputs: the paraphrase and the out-of-scope refusal
for idx in (2, 9):
    r = records[idx]
    print("\nQ:", r["q"])
    print("Docent:", r["answer"])
    print("verdict:", r["verdict"], "|", r["reasoning"])

quality = sum(1 for r in records if r["verdict"] == "correct") / len(records)
print(f"\nJudged quality score: {quality*100:.0f}% correct")

[  correct] What HTTP status code does Meridian return when you exc
[  correct] How many requests per minute does the Pro plan allow?
[  correct] What is the monthly price of the Pro plan?
[  correct] Can you change a database's region after it is created?
[  correct] How long are backups retained on the Scale plan?
[  correct] Which environment variable does the Python SDK read for
[  correct] What does a 401 error mean?
[  correct] How long is the grace period before a deleted database
[  correct] Which plans support point-in-time recovery?
[  correct] Does Meridian support MongoDB-style aggregation pipelin

Q: What is the monthly price of the Pro plan?
Docent: The Pro plan costs $25 per month.
verdict: correct | The candidate states that the Pro plan costs $25 per month, which matches exactly with the reference answer of 25 dollars per month. The phrasing is different but conveys the same factual information accurately.

Q: Does Meridian support MongoDB-style aggregation pipelines?
Docent: I don't have that in the docs.
verdict: correct | The reference indicates this information is not in the docs and that Docent should acknowledge not knowing. The candidate appropriately states they don't have that information in the docs, which aligns with the correct answer.

Judged quality score: 100% correct

Docent went ten for ten: the judge marked every answer correct, for a raw judged quality score of 100%. The two spotlighted rows show the judge doing exactly the work reference metrics couldn’t. On the paraphrase, Docent answered "The Pro plan costs $25 per month." and the judge saw past the different phrasing to the matching fact. On the out-of-scope probe, Docent correctly refused with "I don't have that in the docs." and the judge recognized that a refusal is the right answer here — a distinction exact match would have gotten backwards. This is a genuine live run; re-run it and a phrasing may shift, and on a truly borderline answer a verdict could flip, but on this golden set Docent is solidly correct.

And here is the trap this whole project exists to spring: 100% correct sounds like a triumph, and on its own it is worthless. We have no idea yet whether the judge would have caught a wrong answer, because every answer it saw was right. A judge that blindly stamps “correct” on everything would also report 100% here. Before that number means anything, we have to find out whether the judge can tell right from wrong at all — which is Stage 3.

Stage 3: Calibrate Against Human Labels#

Calibration asks one question: when the judge and a human look at the same answers, how often do they agree? You cannot answer that on the golden run above, because those answers were all correct — an all-correct set has no wrong answers to test the judge’s discrimination, and (as you’ll see) it makes the agreement statistic meaningless. So we build a dedicated calibration set: ten hand-authored candidate answers, deliberately mixed — five genuinely correct, five wrong in instructive ways (a swapped number, the wrong plan’s price, a mislabeled error code, a hallucinated “yes” to the out-of-scope question). Each carries a human gold label of 1 (correct) or 0 (incorrect) — human ground truth, authored by us, not by any model.

The last item is a deliberate landmine: the candidate "About a day." for a question whose reference is "24 hours". A strict human marks it incorrect — it’s vaguer than the reference requires — but a lenient judge may wave it through. That’s exactly the kind of judge–human gap calibration is meant to surface.

from sklearn.metrics import cohen_kappa_score

# HUMAN GOLD LABELS -- authored by hand. 1 = correct, 0 = incorrect. This is ground truth.
CALIB = [
    # --- five genuinely CORRECT candidates (human = 1) ---
    dict(q="How many requests per minute does the Pro plan allow?",
         cand="The Pro plan allows 600 requests per minute.", ref="600 requests per minute", human=1),
    dict(q="What is the monthly price of the Pro plan?",
         cand="It runs 25 dollars a month.", ref="25 dollars per month", human=1),
    dict(q="What does a 401 error mean?",
         cand="A 401 means the API key is missing or invalid.", ref="A missing or invalid API key", human=1),
    dict(q="How long are backups retained on the Scale plan?",
         cand="Scale keeps backups for 90 days.", ref="90 days", human=1),
    dict(q="Does Meridian support MongoDB-style aggregation pipelines?",
         cand="I don't have that in the docs.", ref="(not in the docs -- should refuse)", human=1),
    # --- five INCORRECT candidates (human = 0) ---
    dict(q="How many requests per minute does the Pro plan allow?",
         cand="The Pro plan allows 60 requests per minute.", ref="600 requests per minute", human=0),  # Free plan's number
    dict(q="What is the monthly price of the Pro plan?",
         cand="The Pro plan costs 199 dollars per month.", ref="25 dollars per month", human=0),        # Scale's price
    dict(q="What does a 401 error mean?",
         cand="A 401 means the resource does not exist.", ref="A missing or invalid API key", human=0), # that's 404
    dict(q="Does Meridian support MongoDB-style aggregation pipelines?",
         cand="Yes, Meridian fully supports MongoDB-style aggregation pipelines.",
         ref="(not in the docs -- should refuse)", human=0),                                            # hallucinated yes
    dict(q="How long is the grace period before a deleted database is permanently purged?",
         cand="About a day.", ref="24 hours", human=0),                                                 # too vague -> borderline
]

human = [c["human"] for c in CALIB]
judge = []
for c in CALIB:
    v = judge_answer(c["q"], c["cand"], c["ref"])
    judge.append(1 if v["verdict"] == "correct" else 0)
    c["_verdict"], c["_reason"] = v["verdict"], v["reasoning"]

def agreement(a, b):
    return sum(int(x == y) for x, y in zip(a, b)) / len(a)

def cohen_kappa(a, b):
    n = len(a)
    po = agreement(a, b)                              # observed agreement
    pe = 0.0                                          # agreement expected by chance
    for lab in (0, 1):
        pa = sum(1 for x in a if x == lab) / n
        pb = sum(1 for x in b if x == lab) / n
        pe += pa * pb
    return (po - pe) / (1 - pe)

print("human labels:", human)
print("judge labels:", judge)
agr = agreement(human, judge)
kap = cohen_kappa(human, judge)
print(f"agreement rate : {agr:.2f}")
print(f"Cohen's kappa  : {kap:.2f}   (sklearn check: {cohen_kappa_score(human, judge):.2f})")

# inspect every disagreement by hand -- never average them away
for i in range(len(CALIB)):
    if human[i] != judge[i]:
        c = CALIB[i]
        print("\nDISAGREEMENT:")
        print("  Q:", c["q"])
        print("  candidate:", c["cand"])
        print(f"  human=incorrect  judge={c['_verdict']}")
        print("  judge reasoning:", c["_reason"])

human labels: [1, 1, 1, 1, 1, 0, 0, 0, 0, 0]
judge labels: [1, 1, 1, 1, 1, 0, 0, 0, 0, 1]
agreement rate : 0.90
Cohen's kappa  : 0.80   (sklearn check: 0.80)
DISAGREEMENT:
  Q: How long is the grace period before a deleted database is permanently purged?
  candidate: About a day.
  human=incorrect  judge=correct
  judge reasoning: The candidate states 'about a day' which is essentially equivalent to the reference answer of '24 hours'. Both express the same time period, just using different phrasing. This is acceptable as the candidate conveys the same factual information as the reference without contradiction.

The judge and the human agree on nine of ten, for an agreement rate of 0.90. It cleanly caught all five wrong answers except one — it correctly rejected the swapped rate limit, the wrong price, the mislabeled 401, and the hallucinated “yes” to the out-of-scope question. Because the metric math runs on fixed integer arrays, these numbers are fully reproducible: run the two functions again on the same human and judge lists and you get 0.90 and 0.80 byte for byte, every time — only the live verdicts that fill the arrays can vary.

Agreement alone can flatter a judge, though, which is why we also report Cohen’s kappa = 0.80. Kappa asks how much better than chance the agreement is: with the labels split evenly, two graders would agree about half the time just by guessing, so kappa discounts that baseline. Our raw 0.90 becomes a chance-corrected 0.80 — still strong (kappa above ~0.8 is “almost perfect” by the usual rule of thumb), but honestly lower than the raw rate. This is also the concrete reason the all-correct golden set from Stage 2 can’t be calibrated on: if every human label were 1, the chance-agreement term pe becomes 1, kappa divides by zero, and the statistic is undefined. Kappa only has something to measure when the labels actually vary.

The single disagreement is the most valuable row in the table, and it is why we print every one by hand instead of just reporting the rate. On "About a day." the human said incorrect — vaguer than the docs’ precise “24 hours” — while the judge reasoned “about a day is essentially equivalent to 24 hours” and said correct. Neither is unhinged; they disagree about how much precision a documentation answer owes. That’s a rubric ambiguity, not a bug, and now that you’ve seen it you can decide: tighten the rubric to demand exact durations, or accept that the judge grades a shade more leniently than you would. Averaging that disagreement into a summary statistic would have hidden the one thing worth knowing.

Stage 4: Report the Quality Score With Its Caveat#

Now, and only now, we are allowed to report a number. First we set the bar we committed to before looking at results — a judge we’ll trust needs agreement of at least 0.85 on the calibration set — and check whether this judge clears it. Then we state Docent’s quality score from Stage 2 and its calibration from Stage 3 in the same breath, because a quality score without its calibration is exactly the “confident nonsense” this module opened by warning against.

THRESHOLD = 0.85
trustworthy = agr >= THRESHOLD

print(f"Trust gate: agreement {agr:.2f} vs required {THRESHOLD} -> "
      f"{'TRUSTWORTHY' if trustworthy else 'NOT trustworthy, do not ship this score'}")
print(f"Cohen's kappa: {kap:.2f}")
print()
if trustworthy:
    print("HEADLINE (score + caveat, never one without the other):")
    print(f'  "Docent scores {quality*100:.0f}% correct on the golden set, by a judge that')
    print(f'   agrees with human labels {agr*100:.0f}% of the time (Cohen\'s kappa {kap:.2f})."')
else:
    print("Judge not calibrated enough to trust -- fix the rubric and recalibrate before reporting.")

# the calibration math is deterministic on fixed arrays -- prove it by re-running
print("\nDeterminism re-check (same arrays, twice):")
print("  agreement:", agreement(human, judge), agreement(human, judge))
print("  kappa    :", round(cohen_kappa(human, judge), 4), round(cohen_kappa(human, judge), 4))

Trust gate: agreement 0.90 vs required 0.85 -> TRUSTWORTHY
Cohen's kappa: 0.80

HEADLINE (score + caveat, never one without the other):
  "Docent scores 100% correct on the golden set, by a judge that
   agrees with human labels 90% of the time (Cohen's kappa 0.80)."

Determinism re-check (same arrays, twice):
  agreement: 0.9 0.9
  kappa    : 0.8 0.8

That headline sentence is the entire deliverable of the module, and its shape is the discipline: a score, immediately followed by how far to trust the instrument that produced it. “Docent scores 100% correct” would have been a lie of omission — it invites the reader to believe Docent is flawless, when what we actually know is that a judge which agrees with humans 90% of the time (kappa 0.80) found no errors on this particular golden set. Both halves are load-bearing. The 100% tells you about Docent; the 90%/0.80 tells you how much the 100% is worth. Strip either one and you’re either bragging without evidence or measuring without a ruler.

The re-check makes the reproducibility explicit: the calibration arithmetic is deterministic, so agreement and cohen_kappa return identical values on the same human and judge arrays every single time. What varies between full runs of this notebook is only the live half — Docent’s phrasings and the judge’s verdicts on genuinely borderline items like "About a day." The right mental model is the one from Module 3, now applied to a smarter scorer: let the thing you measure vary, but pin down the ruler and pin down how you validated the ruler.

Practice Exercises#

Exercise 1: Switch the judge from a verdict to a 1-3 rubric score#

A binary correct/incorrect verdict throws away nuance — “correct but vague” and “correct and precise” both score 1. Rewrite the rubric and the JSON contract so the judge returns an integer score of 1 (wrong), 2 (partially correct or too vague), or 3 (fully correct), still with reasoning first. Re-run it on the Stage 3 calibration set and see whether the borderline "About a day." finally lands at 2 instead of a misleading “correct.”

Exercise 2: Add a position-bias control by randomizing, then re-calibrate#

Module 4 warned that judges favor whichever option they see first. Turn one calibration item into a pairwise task — give the judge two candidate answers (one right, one wrong) and ask which is better — then run it twice with the order swapped. If the winner changes with the order, you’ve caught position bias. Randomize the order across a batch and confirm the judge’s agreement with the humans doesn’t depend on presentation.

Exercise 3: Prove an all-correct calibration set can’t be trusted#

Recompute agreement and kappa on a calibration set where every human label is 1 and the judge also returns all 1s. Watch agreement report a perfect 1.00 while kappa blows up or returns nan (division by zero). Write a short guard that refuses to report a judge as calibrated unless the human labels contain both classes and the set has some minimum size.

Summary#

You built a calibrated judge for Docent end to end. You defined the rubric and the judge function — a judge_answer call on claude-haiku-4-5 that returns validated JSON with its reasoning before its verdict, reusing Module 3’s structured-output discipline so a malformed judgement fails loudly. You ran the judge over Docent’s golden set live, including the pricing paraphrase and the out-of-scope refusal, and watched it credit correct paraphrases and reward a well-placed refusal — earning Docent a raw judged score of 100% correct that, on its own, meant nothing. You calibrated the judge against ten hand-authored human gold labels on a mixed set of right and wrong answers, measuring agreement (0.90) and Cohen’s kappa (0.80), confirming the metric math is deterministic, and inspecting the single disagreement — the too-vague "About a day." the judge waved through and a strict human wouldn’t. And you reported the quality score with its caveat attached: “Docent scores 100% correct, by a judge that agrees with humans 90% of the time (kappa 0.80)” — never the score without the calibration.

Key Concepts#

• Reasoning-before-verdict judge — the JSON puts reasoning before verdict so the model commits its thinking before its vote, and the verdict is validated against a closed set of allowed values, borrowing Module 3’s structured-output checks.
• Judged quality score — the fraction of answers a judge marks correct over a golden set; meaningless in isolation because a rubber-stamp judge reports the same number as a discerning one.
• Calibration set — a mixed set of correct and incorrect candidates with human gold labels; the mix is mandatory because a judge’s discrimination can only be tested where there are wrong answers to catch.
• Agreement vs Cohen’s kappa — agreement is the raw fraction of matching labels; kappa discounts chance agreement, is stronger evidence, and is undefined on an all-one-class set — which is precisely why you can’t calibrate on an all-correct golden run.
• Score-with-caveat discipline — you never publish a judge’s quality number without, in the same sentence, stating how well the judge agrees with humans; the caveat is what makes the score usable rather than misleading.

Why This Matters#

An uncalibrated LLM judge is the most dangerous instrument in evaluation precisely because it looks authoritative: it returns clean scores, fluent rationales, and a tidy percentage that a dashboard will happily render as truth. This project gave you the antidote — a repeatable ritual of proving the judge can tell right from wrong on labeled examples before trusting a single one of its verdicts, and of attaching the agreement number to every score so no one downstream mistakes “100% by our judge” for “100%, full stop.” That discipline is what separates a team that ships a judge because it produces numbers from a team that ships a judge because it produces trustworthy numbers — and it generalizes to every automated evaluator you’ll ever stand up, not just this one.

Continue Building Your Skills#

You just turned a module’s worth of judging techniques into one artifact a team can actually trust: a judge that reasons before it votes, returns validated JSON, runs live over Docent’s golden set, and — the part that makes it real — proves itself against human labels before it reports a thing. You measured agreement and Cohen’s kappa, set a trust threshold in advance, inspected the disagreement instead of burying it, and learned to state every quality score in the same breath as the calibration that backs it. That instinct — never a number without its caveat — is what makes model-graded evaluation safe to build on. Next, in Module 5, you’ll turn that evaluative eye on the retrieval that feeds Docent, because a perfect judge can’t fix an answer that was doomed the moment the wrong document was pulled.