Lesson 2 - Rubric & Direct Scoring

Welcome to Rubric & Direct Scoring#

In Lesson 1 you handed a hard-to-grade answer to a model and got back a single verdict: correct or incorrect. That binary is the right tool when there genuinely is one right answer and you just need a yes/no. But most of what Docent – our assistant for the Meridian serverless database docs – produces isn’t a coin flip. An answer can be factually right yet invent a detail the docs never mention. It can be perfectly grounded yet so vague it doesn’t actually help. “Correct or not” flattens all of that into one bit and throws away the shape of the failure.

The instinct is to ask for a number instead: “rate this answer from 1 to 5.” Try it and you’ll watch it drift. Ask twice and you get a 4 then a 3 for the same answer; ask about two different answers and the model’s private, unstated idea of what “4” means has quietly moved between them. A bare numeric scale is noisy because nothing pins the number to anything – the model is inventing the meaning of each score on the fly, differently every time.

A rubric fixes this. Instead of one vague scale, you define a small set of criteria, and for each criterion you spell out what every score level actually means. Now the judge isn’t guessing what a 2 is – it’s matching the answer to an explicit description. Scores stop drifting because they’re anchored, and they become explainable, because you can point at the level the answer landed on and read what that level says. This lesson builds that rubric for Docent and a judge_answer() function that returns reliable, structured, per-criterion scores.

In this lesson, you will:

• See why a bare “rate 1-5” is noisy, and how a rubric anchors each score to an explicit level description
• Design a three-criterion rubric for Docent – Correctness, Groundedness, Helpfulness – each on a 1-3 scale
• Write a scoring prompt that gives the judge the question, answer, and reference, and demands structured JSON with the rationale before the scores
• Parse and validate that JSON, reusing the structured-output discipline from Module 3
• Aggregate per criterion and see the payoff: which dimension is weak, not just an overall grade

Why a rubric beats a bare number#

Picture the naive approach. You send the judge “Here’s a question and Docent’s answer – rate it 1 to 5” and it returns 4. What does 4 mean? The model decided, in that instant, on an implicit standard: maybe it weighed factual accuracy heavily this time, maybe it was lenient about a missing detail, maybe it rewarded a confident tone. Send the same answer again and it might weigh things slightly differently and return 3. Nothing in the prompt fixed the meaning of the scale, so the meaning floats. That variance is real cost: you can’t track a metric that moves when the thing it measures didn’t.

A rubric removes the floating by doing two things at once. First, it decomposes quality into named criteria, so the judge scores one well-defined thing at a time instead of blending everything into a single gestalt number. Second, for each criterion it defines every level in words, so a score is a match against a description rather than a vibe. “Score 3 on Groundedness” isn’t the judge’s mood – it’s the specific claim “every claim is supported by the provided context,” which the judge either can or can’t assert about this answer.

For Docent, three criteria cover the failures that actually happen with a docs assistant:

• Correctness – does the answer match the reference / the docs? This is Lesson 1’s axis, now graded on a scale instead of a bit.
• Groundedness – is every claim supported by the provided context, with nothing invented? An answer can be correct-by-luck while hallucinating, or grounded while wrong; keeping this separate is what lets you catch a confident fabrication.
• Helpfulness – does it actually answer the question, completely and concisely? A grounded, correct-but-evasive “it depends, check the docs” scores low here.

Each runs on a small integer scale – we use 1 to 3 (weak / partial / strong). A short scale is deliberate: the more rungs you add, the more the judge has to split hairs between a 6 and a 7, which reintroduces exactly the noise the rubric was meant to remove. Three well-described levels are far more reliable than ten fuzzy ones.

The scoring prompt#

The judge needs four things in front of it: the question, the answer to grade, the reference (the known-correct answer or supporting context from the docs), and the rubric itself rendered as text. We keep the rubric as a plain Python dict so it’s easy to edit and easy to render into the prompt, then demand a single JSON object back – rationale first, then one integer per criterion.

RUBRIC = {
    "correctness": {
        1: "Contradicts the reference or is factually wrong.",
        2: "Partially correct; some detail wrong, missing, or imprecise.",
        3: "Fully matches the reference; no factual errors.",
    },
    "groundedness": {
        1: "Makes claims not supported by the context; invents facts.",
        2: "Mostly grounded but includes an unsupported detail.",
        3: "Every claim is supported by the provided context.",
    },
    "helpfulness": {
        1: "Does not address the question or is unusable.",
        2: "Addresses the question but is incomplete or padded.",
        3: "Directly and completely answers, concise and clear.",
    },
}

def render_rubric(rubric):
    """Render the rubric dict into the exact text the judge reads."""
    lines = []
    for crit, levels in rubric.items():
        lines.append(f"{crit}:")
        for score in sorted(levels):
            lines.append(f"  {score} = {levels[score]}")
    return "\n".join(lines)

render_rubric turns the dict into the literal criteria-and-levels block the judge sees, so the descriptions in your code are exactly the descriptions the model scores against – no drift between what you wrote and what it read. Now the judge itself:

import anthropic
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

CRITERIA = ("correctness", "groundedness", "helpfulness")

def judge_answer(question, answer, reference, rubric, model="claude-haiku-4-5"):
    prompt = (
        "You are grading a documentation assistant's answer against a rubric. "
        "Score each criterion as an integer using ONLY the level descriptions.\n\n"
        f"Rubric (1-3 per criterion):\n{render_rubric(rubric)}\n\n"
        f"Question:\n{question}\n\n"
        f"Reference answer (ground truth):\n{reference}\n\n"
        f"Assistant answer to grade:\n{answer}\n\n"
        "Respond with a single JSON object and nothing else, with these keys: "
        '"rationale" (one sentence explaining the scores, written BEFORE you decide them), '
        'then "correctness", "groundedness", "helpfulness" (each an integer 1, 2, or 3).'
    )
    msg = client.messages.create(model=model, max_tokens=250,
                                 messages=[{"role": "user", "content": prompt}])
    return msg.content[0].text

Three deliberate choices in that prompt. “Using ONLY the level descriptions” tells the judge to match against the rubric, not its own private standard. Listing "rationale" before the score keys nudges the reasoning-first ordering. And “a single JSON object and nothing else” sets up the structured output we now have to parse – which, as Module 3 hammered home, is where naive judges quietly fall apart.

Parse and validate the verdict#

A judge that returns a paragraph of praise is useless to a metric. We need machine-readable scores, and we need to reject anything that isn’t – a missing criterion, a 5 on a 1-3 scale, a score sent as the string "3". This is the exact structured-output discipline from Module 3, applied to the judge’s reply: strip any markdown fences, parse the JSON, then check every field is present and in range.

import json, re

def extract_json(text):
    """Pull the first JSON object out of a reply, stripping markdown fences."""
    fenced = re.search(r"```(?:json)?\s*(.*?)```", text, re.DOTALL)
    candidate = fenced.group(1) if fenced else text
    start = candidate.find("{")
    end = candidate.rfind("}")
    if start == -1 or end == -1 or end < start:
        return None
    return candidate[start:end + 1]

def parse_scores(raw):
    """Return (scores_dict, status). scores_dict is None if the reply is unusable."""
    payload = extract_json(raw)
    if payload is None:
        return None, "no JSON object found"
    try:
        obj = json.loads(payload)
    except json.JSONDecodeError as e:
        return None, f"JSON did not parse: {e.msg}"
    for c in CRITERIA:
        if c not in obj:
            return None, f"missing criterion {c!r}"
        if not isinstance(obj[c], int) or not (1 <= obj[c] <= 3):
            return None, f"{c} not an integer in 1..3"
    if "rationale" not in obj or not isinstance(obj["rationale"], str):
        return None, "missing or non-string rationale"
    return obj, "ok"

Before we spend a live call, we prove parse_scores accepts what it should and rejects what it shouldn’t, on inputs we control. Because there’s no model in the loop, this block is fully reproducible – the verdicts are identical every run:

CASES = [
    ("good, fenced",
     '```json\n{"rationale": "matches the docs exactly", "correctness": 3, "groundedness": 3, "helpfulness": 3}\n```'),
    ("good, plain",
     '{"rationale": "vague, no source", "correctness": 2, "groundedness": 1, "helpfulness": 1}'),
    ("missing criterion",
     '{"rationale": "ok", "correctness": 3, "groundedness": 3}'),
    ("score out of range",
     '{"rationale": "ok", "correctness": 5, "groundedness": 3, "helpfulness": 3}'),
    ("score as string",
     '{"rationale": "ok", "correctness": "3", "groundedness": 3, "helpfulness": 3}'),
    ("no rationale",
     '{"correctness": 3, "groundedness": 3, "helpfulness": 3}'),
    ("not JSON",
     "The answer looks great, I'd give it top marks."),
]

print("Deterministic parse + validation of judge output")
print("-" * 68)
print(f"{'case':<20}{'ok':<6}detail")
print("-" * 68)
for name, raw in CASES:
    obj, status = parse_scores(raw)
    print(f"{name:<20}{('YES' if obj else 'NO'):<6}{status}")

Deterministic parse + validation of judge output
--------------------------------------------------------------------
case                ok    detail
--------------------------------------------------------------------
good, fenced        YES   ok
good, plain         YES   ok
missing criterion   NO    missing criterion 'helpfulness'
score out of range  NO    correctness not an integer in 1..3
score as string     NO    correctness not an integer in 1..3
no rationale        NO    missing or non-string rationale
not JSON            NO    no JSON object found

The two well-formed replies pass (including the fenced one, because extract_json peels the backticks), and each broken reply is rejected with a precise reason. Re-running gives byte-identical output, because no model is involved:

run2 = [parse_scores(raw)[1] for _, raw in CASES]
print(run2)

['ok', 'ok', "missing criterion 'helpfulness'", 'correctness not an integer in 1..3', 'correctness not an integer in 1..3', 'missing or non-string rationale', 'no JSON object found']

Aggregating the scores#

Per-criterion scores are the point – they tell you which dimension is weak – but you’ll often want a single number per answer too, for sorting or trend lines. The simplest aggregate is the average across criteria. If some dimensions matter more for your product (for a docs assistant, a hallucination is arguably worse than a little verbosity), use a weighted average instead. Both are deterministic arithmetic:

def aggregate(scores, weights=None):
    """Combine per-criterion scores into one number (equal or weighted average)."""
    weights = weights or {c: 1 for c in CRITERIA}
    total_w = sum(weights[c] for c in CRITERIA)
    return sum(scores[c] * weights[c] for c in CRITERIA) / total_w

example = {"correctness": 2, "groundedness": 1, "helpfulness": 1}
print(f"equal-weight avg:     {aggregate(example):.3f}")
w = {"correctness": 3, "groundedness": 2, "helpfulness": 1}
print(f"weighted avg (3/2/1): {aggregate(example, w):.3f}")

equal-weight avg:     1.333
weighted avg (3/2/1): 1.500

The weighted number is higher here because this answer’s one relative strength – correctness at 2 – carries more weight, while its weak groundedness and helpfulness count for less. Weighting is a product decision, not a statistical one: you’re encoding what your users would trade off. But whatever you aggregate to, keep reporting the per-criterion breakdown. A single 1.33 hides the story; correctness=2, groundedness=1, helpfulness=1 tells you exactly where this answer went wrong.

Running the judge live on Docent#

Now the real thing. We grade four Docent answers spanning the failure modes a docs assistant actually hits: a strong answer that nails the fact, a vague/partial one that’s roughly right but imprecise, a hallucinated one that confidently invents a feature, and a wrong-fact one that states a clean but incorrect number. The references come from the course’s golden Q&A set. Because the judge is a live model, treat these numbers as a real recorded run – yours will be close but not byte-identical on re-run.

CASES = [
    dict(label="strong",
         question="How many requests per minute does the Pro plan allow?",
         reference="600 requests per minute",
         answer="The Pro plan allows 600 requests per minute."),
    dict(label="partial",
         question="How long are backups retained on the Scale plan?",
         reference="90 days",
         answer="Backups are retained for a while on the Scale plan, roughly a few months."),
    dict(label="hallucinated",
         question="Can you change a database's region after it is created?",
         reference="No; the region is fixed at creation and you must export and re-import into a new database to move regions.",
         answer="Yes. Open Settings > Regions and pick a new region; Meridian migrates the data live with no downtime."),
    dict(label="wrong-fact",
         question="How much does the Pro plan cost per month?",
         reference="25 dollars per month",
         answer="The Pro plan costs 30 dollars per month."),
]

print("Live rubric judging over Docent answers (claude-haiku-4-5)")
print("=" * 74)
for c in CASES:
    raw = judge_answer(c["question"], c["answer"], c["reference"], RUBRIC)
    scores, status = parse_scores(raw)
    print(f"\n[{c['label']}] {c['answer'][:60]}")
    if scores is None:
        print(f"  PARSE FAILED: {status}")
        continue
    print(f"  correctness={scores['correctness']}  groundedness={scores['groundedness']}"
          f"  helpfulness={scores['helpfulness']}  avg={aggregate(scores):.2f}")
    print(f"  rationale: {scores['rationale']}")

Live rubric judging over Docent answers (claude-haiku-4-5)
==========================================================================

[strong] The Pro plan allows 600 requests per minute.
  correctness=3  groundedness=3  helpfulness=3  avg=3.00
  rationale: The answer directly states '600 requests per minute' which exactly matches the reference answer, is fully supported by the context provided, and concisely addresses the question without any extraneous information.

[partial] Backups are retained for a while on the Scale plan, roughly a
  correctness=2  groundedness=1  helpfulness=1  avg=1.33
  rationale: The answer is vague and imprecise ('a while', 'roughly a few months') when the reference specifies exactly 90 days, making it partially correct but lacking the specific detail needed, unsupported by any contextual reference provided, and unhelpful due to its imprecision.

[hallucinated] Yes. Open Settings > Regions and pick a new region; Meridian
  correctness=1  groundedness=1  helpfulness=1  avg=1.00
  rationale: The answer directly contradicts the reference material by claiming regions can be changed after creation with live migration, when the correct answer states regions are fixed and require export/re-import.

[wrong-fact] The Pro plan costs 30 dollars per month.
  correctness=1  groundedness=1  helpfulness=1  avg=1.00
  rationale: The answer provides a specific price that directly contradicts the reference answer ($30 vs $25), making it factually incorrect and therefore not helpful, though it does attempt to address the question asked.

Read down the per-criterion columns and the rubric earns its keep. The strong answer is a clean 3/3/3. The partial answer is the interesting one: correctness=2 (it’s in the right ballpark) but groundedness=1 and helpfulness=1 – a bare “rate 1-5” would have collapsed that into a single mushy number, whereas the rubric tells you precisely that the answer’s problem is vagueness and lack of support, not being flat-out wrong. The hallucinated answer bottoms out at 1 on every axis, and its rationale names the exact fabrication (“claiming regions can be changed… with live migration”). That’s the payoff of per-criterion scoring: it exposes which dimension failed, and the rationale gives you the receipt.

Be honest about variation. Run this again and the vague answer might come back correctness=2, groundedness=2 instead of 1, or a rationale worded differently – the judge is a model, and its scores wobble by a level on the genuinely borderline cases. That wobble is small and the ranking is stable (strong > partial > hallucinated holds every run), which is exactly why the deterministic parse/aggregate code around it is worth having: the arithmetic never drifts, only the judgement it operates on does. Later in this module you’ll measure that wobble by calibrating the judge against human labels; for now, know it’s there and don’t over-read a single-level difference.

Practice Exercises#

Exercise 1: Add a fourth criterion#

Docent should refuse questions the docs can’t answer (the out-of-scope probe “Does Meridian support MongoDB-style aggregation pipelines?”). Add a fourth criterion, conciseness or refusal_appropriateness, to RUBRIC with three explicit level descriptions, update CRITERIA, and re-run the live judge on the four answers. Does adding a criterion change the scores on the existing ones, and does parse_scores still validate the now-four-key JSON?

Exercise 2: Measure the judge’s run-to-run wobble#

The lesson claims scores wobble by about a level on borderline answers but the ranking is stable. Test it: pick the partial case, call judge_answer on it five times, collect the correctness/groundedness/helpfulness scores, and print the min, max, and mean of each. How much does each criterion move across the five runs?

Exercise 3: Weight groundedness to punish hallucination#

For a docs assistant, a confident hallucination is the most dangerous failure. Build a weights dict that makes groundedness count double the others, compute the weighted aggregate for all four live answers, and compare the ranking to the equal-weight average. Does emphasizing groundedness change which answer looks worst?

Summary#

A bare “rate this 1-5” is noisy because nothing anchors the number – the judge invents the scale’s meaning on every call, so scores drift run to run and tell you nothing about what was wrong. A rubric fixes both problems by decomposing quality into named criteria and defining what each score level means, so the judge matches an answer to an explicit description instead of guessing. You built a three-criterion rubric for Docent – Correctness, Groundedness, Helpfulness, each on a 1-3 scale – and a judge_answer() that calls claude-haiku-4-5 with the question, answer, reference, and rendered rubric, demanding a single JSON object with the rationale written before the scores so the numbers fall out of reasoning rather than a snap judgement. You reused Module 3’s structured-output discipline to parse_scores – stripping fences, parsing JSON, and rejecting any reply with a missing criterion, an out-of-range value, a mistyped score, or no rationale – proving it reproducibly on seven hand-crafted cases. Live on four Docent answers, the rubric did what a single number can’t: the strong answer scored 3/3/3, the hallucinated and wrong-fact answers bottomed out at 1/1/1 with rationales naming the fabrication, and the vague answer split into correctness=2, groundedness=1, helpfulness=1 – exposing that its problem was imprecision, not being flat-out wrong.

Key Concepts#

• Rubric – a set of named criteria, each with explicit descriptions for every score level; it anchors scores so they’re consistent and explainable instead of drifting.
• Direct (pointwise) scoring – grading a single answer on an absolute scale per criterion, as opposed to comparing two answers (next lesson’s pairwise approach).
• Per-criterion decomposition – Correctness, Groundedness, Helpfulness scored separately, so a weak dimension is visible rather than blended into one number.
• Rationale-before-score – asking the judge to reason before it grades; the reasoning improves reliability and gives you an auditable trail for each score.
• Structured, validated output – the judge returns JSON that you parse and validate (fields present, integers in range, rationale a string); an unparseable verdict is a signal, not a crash.
• Aggregation – combining criteria into one number via an equal or weighted average for sorting/trends, while still always reporting the per-criterion breakdown.

Why This Matters#

Direct rubric scoring is the workhorse of LLM evaluation: it’s how teams turn “is this open-ended answer any good?” into numbers they can track on a dashboard, gate a deploy on, and slice by failure type. The two design choices in this lesson are the ones that separate a metric you can trust from one you can’t. Defining every score level is what makes the number stable – without it you’re charting the judge’s mood. Scoring per criterion is what makes the number actionable – a drop in an aggregate score means nothing until you can see it was Groundedness that fell, which points at retrieval or hallucination rather than, say, verbosity. And building the parse-and-validate layer around the judge is what keeps a fleet of judgements from silently corrupting on the one reply that came back as prose. You now have a judge that produces reliable, explainable, per-criterion scores for Docent. In the next lesson you’ll see that when you need to compare two candidate answers, asking “which is better?” is often more reliable than scoring each in isolation – pairwise preference.

Continue Building Your Skills#

You can now get a reliable, explainable score out of an LLM judge instead of a noisy guess: a rubric that names the criteria and defines each level, a judge_answer() that returns structured JSON with the rationale before the scores, a parse-and-validate step that rejects any malformed verdict, and per-criterion reporting that shows which dimension is weak – Groundedness collapsing on the hallucinated answer while the strong one holds a clean 3/3/3. That’s direct scoring: grading one answer on an absolute scale. But absolute scales have a limit – calibrating “what does a 2 mean” is genuinely hard, and judges are more consistent when they compare than when they measure. The next lesson turns to pairwise comparison: hand the judge two candidate answers to the same question and ask which is better, a framing that sidesteps scale calibration and, as you’ll see, often agrees with humans more closely – while introducing a new hazard, position bias, that the lesson after that confronts head-on.