Lesson 4 - Calibrating & Trusting the Judge

Welcome to Calibrating & Trusting the Judge#

In the last three lessons you taught a model to grade Docent’s answers about Meridian — a rubric and direct scores in Lesson 2, pairwise preference in Lesson 3. Every one of those techniques quietly assumed the judge is right. This lesson attacks that assumption head on, and it is the most important lesson in the module: a judge is itself a model, and a model can be confidently, systematically wrong. A judge that scores everything a 4, or always picks the first option, or rewards the longest answer, will hand you a beautiful dashboard full of numbers that mean nothing. Before you let a judge gate a deploy, you have to earn the right to trust it — and “earn” here means measure.

The tool for that measurement is older than LLMs: treat the judge as a classifier and compare it against a small set of human judgements you trust. If the judge and a human agree on 9 of 10 answers, you can lean on it for the other thousands. If they agree on 6 of 10, the judge is barely better than a coin flip and you fix it or throw it out. Along the way you’ll meet the specific biases that make naive judges fail, and you’ll see — with real claude-haiku-4-5 verdicts — a naive judge score worse than chance against humans and a grounded judge score perfectly, on the exact same answers.

In this lesson, you will:

• Name the biases that make a naive judge untrustworthy: position, verbosity, self-preference, leniency/severity, and prompt sensitivity
• Build a small human-labeled gold set of Docent answers and treat the judge as a classifier against it
• Compute raw agreement and Cohen’s kappa (by hand and with sklearn) between judge and human, and read what the numbers mean
• Fix a failing judge by grounding its prompt, and probe for verbosity bias directly

A judge is a classifier you have not tested yet#

When you ask a model “is this answer correct — yes or no?”, you have built a binary classifier. It takes an input (a question and an answer) and emits a label (correct / incorrect). We already have a whole discipline for deciding whether a classifier is any good: run it on examples where you know the true label, and see how often it matches. The only new wrinkle is where the true labels come from. For a spam filter they come from a labeled dataset; for a judge they come from humans reading the answers and recording their own verdict. Those human verdicts are the ground truth — not because humans are infallible, but because human judgement is the thing the judge is trying to approximate, so it is the standard the judge must be measured against.

This reframing is the whole lesson. You do not trust a judge because it sounds authoritative or because it is a big model. You trust it because you checked it against humans on a sample and it agreed often enough. Everything else — better rubrics, reasoning before verdict, ensembling — is a way to raise that agreement number. But you cannot improve what you refuse to measure, so measurement comes first.

The biases that break a naive judge#

Before we measure, it helps to know what we are measuring for. Judges fail in patterned, predictable ways, and each pattern is a reason the judge might disagree with a human:

• Position bias — in a pairwise comparison the judge favors whichever option it sees first, regardless of quality. You met this in Lesson 3, and the fix was to randomize order (or judge both orders and average).
• Verbosity bias — the judge rewards longer, more elaborate answers even when the extra words add nothing or introduce errors. A padded wrong answer can outscore a terse right one.
• Self-preference — the judge favors answers written in its own style, or generated by its own model family, over equally good answers that read differently.
• Leniency and severity — the judge’s scores cluster. A lenient judge marks almost everything correct (and misses real errors); a severe judge marks almost everything wrong (and punishes good answers). Either way the scores stop discriminating.
• Prompt sensitivity — small wording changes in the judge’s instructions swing its verdicts. “Is this a good answer?” and “Grade this answer against these facts” are different judges, and they will not agree with each other, let alone with a human.

You do not need to memorize a mitigation for each one right now. The point is that these biases are real, they are common, and the only way to know whether your judge on your task has them is to calibrate against humans and look at where it disagrees.

Build the human-labeled gold set#

Here is our calibration set: ten Docent answers to Meridian questions, each with a human gold label we authored — 1 if a human grader judged the answer correct, 0 if incorrect. These labels are the ground truth for this exercise; we wrote them by reading each answer against the Meridian docs. The set is deliberately mixed: some answers are clean and correct, some are flatly wrong, and one (item 0) is the tricky kind — the right status code wrapped around a fabricated detail (“wait 60 seconds”), which a careful human marks incorrect.

# Each answer carries a HUMAN gold label authored by us (the ground truth for calibration).
# human = 1 -> a human grader judged the answer CORRECT, 0 -> INCORRECT.
GOLD = [
    dict(q="What HTTP status code does Meridian return when you exceed the rate limit?",
         a="It returns a 429 Too Many Requests, and you should wait 60 seconds before retrying.",
         human=0),  # 429 is right, but "60 seconds" is fabricated; the docs give a Retry-After header
    dict(q="How many requests per minute does the Pro plan allow?",
         a="The Pro plan allows 600 requests per minute.", human=1),
    dict(q="How much does the Pro plan cost per month?",
         a="The Pro plan costs 25 dollars per month.", human=1),
    dict(q="Can you change a database's region after it is created?",
         a="Yes, you can switch a database's region at any time from the dashboard.",
         human=0),  # region is fixed at creation; this is wrong
    dict(q="How long are backups retained on the Scale plan?",
         a="Backups are retained for 30 days on the Scale plan.",
         human=0),  # Scale is 90 days; 30 is the Pro figure
    dict(q="Which environment variable does the Python SDK read for the API key?",
         a="It reads the MERIDIAN_API_KEY environment variable.", human=1),
    dict(q="What does a 401 error mean?",
         a="A 401 error means the key is valid but lacks permission for the resource.",
         human=0),  # that describes 403, not 401
    dict(q="How long is the grace period before a deleted database is permanently purged?",
         a="There is a 24-hour grace period before the data is permanently purged.", human=1),
    dict(q="Which plans support point-in-time recovery?",
         a="Point-in-time recovery is available only on the Scale plan.", human=1),
    dict(q="Does Meridian support MongoDB-style aggregation pipelines?",
         a="I don't have that in the docs.", human=1),  # correct behavior: refuse the out-of-scope question
]

human_labels = [g["human"] for g in GOLD]
print("human gold labels:", human_labels)
print("correct:", human_labels.count(1), " incorrect:", human_labels.count(0))

human gold labels: [0, 1, 1, 0, 0, 1, 0, 1, 1, 1]
correct: 6  incorrect: 4

Six answers a human calls correct, four incorrect — a realistic mix, not all-good or all-bad, so a judge cannot get a high score just by saying “correct” every time. That balance matters: if nine of ten answers were correct, a lazy judge that always says “correct” would already agree 90% of the time and look great while being useless. A mixed set forces the judge to actually discriminate, which is exactly what we want to measure.

Run a naive judge and score it against the humans#

Now the judge. We start with the kind of judge people reach for first: a one-line prompt with no reference facts — just “is this a good answer?” It reads plausible, and that is precisely the trap. We run it live over the ten answers with claude-haiku-4-5 (reading the API key from the environment; never hard-coded), collect a 0/1 verdict for each, and compare to the human labels.

import anthropic
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

def naive_judge(q, a, model="claude-haiku-4-5"):
    prompt = (f"Is this a good answer to the question? Reply CORRECT or INCORRECT.\n\n"
              f"Question: {q}\nAnswer: {a}")
    msg = client.messages.create(model=model, max_tokens=5,
                                 messages=[{"role": "user", "content": prompt}])
    v = msg.content[0].text.strip().upper()
    return 0 if v.startswith("INCORRECT") else (1 if v.startswith("CORRECT") else 0)

naive_labels = [naive_judge(g["q"], g["a"]) for g in GOLD]

print("idx  human  judge  match")
for i, (g, j) in enumerate(zip(GOLD, naive_labels)):
    print(f"{i:>3}    {g['human']}      {j}     {'ok' if g['human'] == j else 'DISAGREE'}")

idx  human  judge  match
  0    0      0     ok
  1    1      0     DISAGREE
  2    1      1     ok
  3    0      0     ok
  4    0      1     DISAGREE
  5    1      0     DISAGREE
  6    0      0     ok
  7    1      0     DISAGREE
  8    1      0     DISAGREE
  9    1      0     DISAGREE

This is one real run — the judge samples, so a re-run may flip a verdict or two — but the shape is stable and it is alarming. The naive judge disagrees with the human on six of ten answers, and look at how it disagrees. It marks five genuinely correct answers (items 1, 5, 7, 8, 9) as INCORRECT — that is severity bias: with no reference facts to check against, the judge got suspicious of short, confident answers and rejected them. And on item 4 — “Backups are retained for 30 days on the Scale plan,” which is flatly wrong (Scale is 90 days) — it said CORRECT, because the answer sounds authoritative and the judge had no facts to catch the error. A judge that rejects right answers and accepts wrong ones is not a mild problem. But “it feels bad” is not a measurement. Let’s put a number on it.

Measure agreement and Cohen’s kappa#

Two numbers turn “the judge disagreed a lot” into something you can track and act on. The first is raw agreement: the fraction of items where judge and human gave the same label. It is trivial to compute and easy to explain. The second is Cohen’s kappa, which corrects raw agreement for the agreement you’d expect by chance — because two labelers who both say “correct” most of the time will agree often even if neither is looking at the answer. Kappa asks: how much better than random luck is this agreement?

The formula is

where $p_o$ is the observed agreement (the raw rate) and $p_e$ is the agreement expected by chance, computed from how often each labeler uses each label. Kappa of 1.0 is perfect agreement, 0.0 is exactly chance, and — importantly — a negative kappa means the two labelers agree less than random labeling would. We compute both numbers by hand (so the formula is not a black box) and cross-check kappa against sklearn.

Crucially, this computation is deterministic: given the two fixed label arrays, it returns the same numbers every run. So we freeze the judge verdicts we just recorded into a plain list and compute on that — the live model is the stochastic half, the metric is the reproducible half.

from sklearn.metrics import cohen_kappa_score

human  = [0, 1, 1, 0, 0, 1, 0, 1, 1, 1]   # human gold labels (authored by us)
naive  = [0, 0, 1, 0, 1, 0, 0, 0, 0, 0]   # recorded naive-judge verdicts from the run above

def agreement_rate(a, b):
    return sum(int(x == y) for x, y in zip(a, b)) / len(a)

def cohen_kappa(a, b):                       # kappa from scratch
    n = len(a)
    p_o = agreement_rate(a, b)               # observed agreement
    p_e = sum((sum(1 for x in a if x == lab) / n) *   # chance agreement:
              (sum(1 for x in b if x == lab) / n)     #   sum over labels of
              for lab in set(a) | set(b))             #   P_a(lab) * P_b(lab)
    return (p_o - p_e) / (1 - p_e)

n = len(human)
agree = sum(int(h == j) for h, j in zip(human, naive))
print(f"agreement rate = {agree}/{n} = {agreement_rate(human, naive):.2f}")
print(f"cohen kappa (by hand) = {cohen_kappa(human, naive):.2f}")
print(f"cohen kappa (sklearn) = {cohen_kappa_score(human, naive):.2f}")

agreement rate = 4/10 = 0.40
cohen kappa (by hand) = -0.07
cohen kappa (sklearn) = -0.07

The verdict on the verdict-giver: agreement 0.40, kappa −0.07. The naive judge agrees with humans on fewer than half the answers, and its kappa is negative — it is very slightly worse than labeling by coin flip. This judge is not a weak signal you could smooth over with more samples; it is anti-correlated with the truth on this set. If you had trusted it, it would have failed correct answers and passed a wrong backup-retention claim straight into your “quality” dashboard. Our hand-rolled kappa matches sklearn to the decimal, and because both inputs are fixed lists, re-running this block prints the identical 0.40 / −0.07 / −0.07 every time — the metric is the deterministic ruler even though the judge that produced naive was not.

Fix the judge, then re-measure#

A kappa of −0.07 is not a reason to give up on LLM judging — it is a reason to fix this judge. The naive judge’s core problem is obvious once named: it had no ground truth to grade against, so it fell back on vibes and got severe. The fix is the same discipline from Lesson 2 — give the judge a clear rubric and the reference facts — turned into a better prompt. Then we do the only thing that proves a fix worked: re-measure against the same human labels.

FACTS = ("Rate limit exceeded -> HTTP 429, returned with a Retry-After header giving the seconds to wait. "
         "Pro plan: 600 req/min, costs 25 dollars/month. "
         "Region is fixed at creation and cannot be changed (export/re-import to move). "
         "Backups retained: Free 7 days, Pro 30 days, Scale 90 days. "
         "Python SDK reads MERIDIAN_API_KEY. 401 = missing or invalid API key; "
         "403 = valid key that lacks permission. Deleting a database has a 24-hour grace period "
         "before permanent purge. Point-in-time recovery is Scale-plan only. "
         "The docs do not cover MongoDB-style aggregation pipelines.")

def grounded_judge(q, a, model="claude-haiku-4-5"):
    prompt = (f"You grade a documentation assistant's answer for factual correctness against the "
              f"reference facts. Reply with exactly one word: CORRECT or INCORRECT.\n\n"
              f"Reference facts:\n{FACTS}\n\nQuestion: {q}\nAssistant answer: {a}\n\nVerdict:")
    msg = client.messages.create(model=model, max_tokens=5,
                                 messages=[{"role": "user", "content": prompt}])
    v = msg.content[0].text.strip().upper()
    return 0 if v.startswith("INCORRECT") else (1 if v.startswith("CORRECT") else 0)

grounded_labels = [grounded_judge(g["q"], g["a"]) for g in GOLD]

agree = sum(int(h == j) for h, j in zip(human_labels, grounded_labels))
print("grounded verdicts:", grounded_labels)
print(f"agreement rate = {agree}/{len(human_labels)} = {agree/len(human_labels):.2f}")
print(f"cohen kappa    = {cohen_kappa_score(human_labels, grounded_labels):.2f}")

grounded verdicts: [0, 1, 1, 0, 0, 1, 0, 1, 1, 1]
agreement rate = 10/10 = 1.00
cohen kappa    = 1.00

The same model, the same ten answers, the same human labels — and now agreement 1.00, kappa 1.00. Grounding the judge in the reference facts and a crisp rubric moved every single item onto the diagonal. It correctly failed the fabricated “60 seconds” retry claim (item 0), the swap of 90 days for 30 (item 4), and the 401-described-as-403 error (item 6), while accepting the six genuinely correct answers and the correct refusal on the out-of-scope MongoDB question. This is a live run, so a re-run might slip to 9/10 on a genuinely borderline phrasing — but the leap from 0.40 to near-perfect is not noise, it is the rubric doing its job. The figure below shows both judges as confusion matrices side by side: the naive judge scatters mass off the diagonal, the grounded judge concentrates it on.

Probe for verbosity bias directly#

Agreement against a gold set catches biases in aggregate, but you can also probe for a specific bias with a targeted experiment. Verbosity bias is the classic one, and the probe is elegantly simple: hold the fact constant and vary only the length. Score the same correct answer twice — once terse, once padded with fluff — and see whether the padding buys a higher score. If it does, your judge is paying for words, not correctness.

vq = "How many requests per minute does the Pro plan allow?"
terse = "600 requests per minute."
verbose = ("Great question! According to the Meridian rate limiting documentation, the Pro plan "
           "is quite generous and allows a full 600 requests every single minute, which should give "
           "you plenty of headroom for most production workloads and typical traffic spikes.")

def score_1to5(q, a, model="claude-haiku-4-5"):
    import re
    prompt = (f"Score this documentation answer for quality from 1 to 5 (5 = best). "
              f"Reply with only the integer.\n\nReference facts:\n{FACTS}\n\n"
              f"Question: {q}\nAnswer: {a}\n\nScore:")
    msg = client.messages.create(model=model, max_tokens=5,
                                 messages=[{"role": "user", "content": prompt}])
    m = re.search(r"[1-5]", msg.content[0].text)
    return int(m.group()) if m else None

print("verbosity probe (same fact: 600 req/min):")
print(f"  terse   ({len(terse):>3} chars) -> {score_1to5(vq, terse)}")
print(f"  verbose ({len(verbose):>3} chars) -> {score_1to5(vq, verbose)}")

verbosity probe (same fact: 600 req/min):
  terse   ( 24 chars) -> 5
  verbose (252 chars) -> 5

On this run the grounded claude-haiku-4-5 judge scored both the terse and the ten-times-longer answer a 5 — no verbosity bias detected. That is the good outcome, and it is worth stating why we ran the probe anyway: you only know the judge is clean because you checked, not because you assumed. Weaker or older judge models frequently inflate the padded version by a point or two, and a judge that does will silently reward Docent for waffling — nudging your whole system toward longer, wordier answers over time. Keep this probe (and a position-bias twin from Lesson 3) as a tiny standing test suite you run whenever you change the judge prompt or model; a null result today does not guarantee a null result after your next “harmless” tweak.

Practice Exercises#

Exercise 1: Score a second human labeler and compute inter-annotator kappa#

Kappa is not just for judge-vs-human — it is the standard way to measure whether two humans agree, which sets the ceiling on how well any judge can possibly do. Author a second column of human labels for the same ten answers (imagine a stricter colleague who fails item 8 too), and compute the kappa between the two humans. Then compare it to the grounded judge’s kappa against the first human.

Exercise 2: Build the confusion matrix and label the two error types#

Raw agreement collapses two very different mistakes into one number. Write a function that returns the 2×2 confusion matrix (human-correct/judge-correct, human-correct/judge-incorrect, human-wrong/judge-correct, human-wrong/judge-incorrect) for a judge, and print it for both naive and grounded. Then name which off-diagonal cell is leniency (letting a wrong answer pass) and which is severity (failing a right one).

Exercise 3: Ensemble three judge calls with a majority vote#

A single judge verdict can be flaky on borderline items. Wrap grounded_judge so it calls the model three times for the same answer and returns the majority label, then re-run the calibration and compare its agreement and kappa to the single-call judge. Note how many total live calls this costs so the trade-off is explicit.

Summary#

You learned the discipline that makes an LLM judge safe to rely on: calibrate it against human labels before you trust it. You reframed the judge as a classifier and scored it against a ten-item human gold set for Docent, using raw agreement and Cohen’s kappa — which you implemented by hand and cross-checked against sklearn. A naive judge with no reference facts scored 0.40 agreement and kappa −0.07 — worse than a coin flip, failing correct answers out of severity and passing a wrong backup claim. Grounding the judge in the reference facts and a clear rubric lifted the same model on the same answers to 1.00 agreement and kappa 1.00, and a verbosity probe confirmed the grounded judge scored a terse and a padded version of the same fact identically. The metric code was deterministic throughout — reproducible on the frozen label arrays — while the live judge was honestly stochastic.

Key Concepts#

• Judge as classifier — a judge emits labels, so you evaluate it exactly like any classifier: run it on items with known (human) labels and measure how often it matches.
• Human gold set — a small, hand-labeled sample (10 here, typically 30–100) that is the ground truth for calibration; you label just enough to trust the judge on everything you didn’t label.
• Agreement rate — the fraction of items where judge and human give the same verdict; simple and interpretable, but inflated on skewed sets.
• Cohen’s kappa — agreement corrected for chance, $\kappa = (p_o - p_e)/(1 - p_e)$; 1.0 perfect, 0.0 chance, negative worse-than-chance; honest where raw agreement lies.
• Judge biases — position, verbosity, self-preference, leniency/severity, and prompt sensitivity are the patterned ways judges disagree with humans; grounding the prompt, requiring reasoning, randomizing order, and ensembling are the levers that raise agreement.

Why This Matters#

An uncalibrated judge is worse than no judge, because it launders opinion into numbers and numbers earn unearned trust — a dashboard reading “94% correct” from a lenient judge will get shipped on, and the errors it missed ship with it. This lesson gives you the one habit that prevents that: never quote a judge’s score without a calibration number sitting next to it. When you can say “this judge agrees with our humans at kappa 0.85 on a 50-item gold set,” the judge’s verdicts on ten thousand unlabeled answers become genuinely load-bearing — you have measured the ruler before trusting the measurements. And when the kappa is low, you now know it is fixable: ground the prompt, sharpen the rubric, ensemble the calls, and re-measure until the judge earns the trust you were about to give it for free.

Continue Building Your Skills#

You just crossed the line that separates people who use LLM judges from people who trust them responsibly. A judge is a model, and a model can be systematically wrong — so you treated it as a classifier, held it up against a small set of human judgements, and let raw agreement and Cohen’s kappa tell you, in a number, whether to believe it. You watched a naive judge score worse than chance and a grounded one score perfectly on the very same answers, and you probed a specific bias head-on. In the guided project that closes this module, you’ll build a calibrated judge for Docent from scratch — gold set, agreement metrics, rubric iteration, and a defensible trust threshold — the judge you’d actually be willing to put in front of a deploy.