Lesson 5 - Guided Project: A Reference-Metric Scorecard

Welcome to the Guided Project#

This is the capstone for Deterministic & Reference Metrics, and it pulls the whole module into one artifact you can actually run: a scorecard. Over the last four lessons you met exact match and the normalization that makes it fair, token-level F1 for partial credit, ROUGE-L for surface overlap, and structured-output checks. Individually each metric tells a slanted story. Put side by side over the same answers, they tell a much more honest one — and the disagreement between them is the most useful signal of all.

The four stages mirror how a team stands up a reference-metric eval. You assemble the pieces — a golden set and the three scorers behind one shared normalizer; you run Docent live over the golden questions and collect its real answers; you score every answer on every metric into a per-item table and then aggregate to a single scorecard; and you read the scorecard to understand why the numbers disagree, which to trust for Docent’s short factual answers, and exactly where all three quietly fail. The scoring code is deterministic — the same answers produce the same numbers every run — while the live model is not, and seeing both halves is the point.

By the end of this project, you will be able to:

• Assemble a golden set and three reference metrics — normalized exact match, token F1, and ROUGE-L (via rouge-score) — behind one SQuAD-style normalizer
• Run Docent live over the golden set and collect its real answers, understanding why they vary run to run
• Build a per-item scorecard (question, EM, F1, ROUGE-L) and aggregate it to mean EM, mean F1, and mean ROUGE-L
• Read the scorecard critically: why the metrics disagree, which to trust for short factual answers, and where all three fall short

Stage 1: Assemble the Pieces#

A scorecard needs two ingredients: a golden set of questions with known-correct references, and a set of metric functions that turn a (prediction, reference) pair into a number. We reuse the module’s eight factual golden questions — each has a short reference answer drawn straight from Meridian’s docs — and the three scorers you built across the module. The one thing they all share is normalization: the SQuAD-style cleanup that lowercases, strips punctuation and articles, and collapses whitespace, so that "HTTP 429." and "429" aren’t judged different over a period and a word.

import re, string
from collections import Counter
from rouge_score import rouge_scorer

# --- the golden set: 8 factual questions with short reference answers ---
GOLDEN = [
    dict(q="What HTTP status code does Meridian return when you exceed the rate limit?",
         ref="429", doc="ratelimits"),
    dict(q="How many requests per minute does the Pro plan allow?",
         ref="600 requests per minute", doc="ratelimits"),
    dict(q="How much does the Pro plan cost per month?",
         ref="25 dollars per month", doc="plans"),
    dict(q="How long are backups retained on the Scale plan?",
         ref="90 days", doc="backups"),
    dict(q="Which environment variable does the Python SDK read for the API key?",
         ref="MERIDIAN_API_KEY", doc="sdks"),
    dict(q="What does a 401 error mean?",
         ref="A missing or invalid API key", doc="errors"),
    dict(q="How long is the grace period before a deleted database is permanently purged?",
         ref="24 hours", doc="deletion"),
    dict(q="Which plans support point-in-time recovery?",
         ref="Only the Scale plan", doc="backups"),
]

# --- one shared SQuAD-style normalizer ---
def normalize(s):
    s = s.lower()
    s = "".join(ch for ch in s if ch not in set(string.punctuation))
    s = re.sub(r"\b(a|an|the)\b", " ", s)   # drop articles
    return " ".join(s.split())              # collapse whitespace

# --- metric 1: normalized exact match (all-or-nothing) ---
def exact_match(pred, ref):
    return int(normalize(pred) == normalize(ref))

# --- metric 2: token-level F1 (partial credit for word overlap) ---
def f1_score(pred, ref):
    p, r = normalize(pred).split(), normalize(ref).split()
    if not p or not r:
        return float(p == r)
    common = Counter(p) & Counter(r)
    overlap = sum(common.values())
    if overlap == 0:
        return 0.0
    precision, recall = overlap / len(p), overlap / len(r)
    return 2 * precision * recall / (precision + recall)

# --- metric 3: ROUGE-L (longest-common-subsequence overlap) via the library ---
_rouge = rouge_scorer.RougeScorer(["rougeL"], use_stemmer=True)
def rouge_l(pred, ref):
    return _rouge.score(ref, pred)["rougeL"].fmeasure

# sanity-check all three on a few fixed (prediction, reference) pairs — deterministic
demo = [
    ("429", "429"),
    ("The Pro plan allows 600 requests per minute.", "600 requests per minute"),
    ("It costs 25 dollars per month.", "25 dollars per month"),
    ("It means the API key is missing or invalid.", "A missing or invalid API key"),
]
print(f"{'reference':<26} {'EM':>3} {'F1':>6} {'ROUGE-L':>8}")
for pred, ref in demo:
    print(f"{ref[:25]:<26} {exact_match(pred, ref):>3} "
          f"{f1_score(pred, ref):>6.2f} {rouge_l(pred, ref):>8.2f}")

reference                   EM     F1  ROUGE-L
429                          1   1.00     1.00
600 requests per minute      0   0.73     0.67
25 dollars per month         0   0.80     0.80
A missing or invalid API     0   0.77     0.40

The sanity check already previews the whole module in four rows. When the prediction is the reference ("429"), all three metrics agree perfectly at 1.00. The moment the model wraps the fact in a sentence — "The Pro plan allows 600 requests per minute." — exact match collapses to 0 even though the answer is completely correct, while F1 and ROUGE-L award partial credit (0.73/0.67) for the words that do overlap. The last row shows F1 and ROUGE-L can themselves disagree: F1 counts the shared bag of words (0.77), but ROUGE-L insists they appear in the same order and drops to 0.40 because “missing or invalid” and “the API key” are reordered. Same answer, three different verdicts — that spread is exactly what a scorecard exposes. And because none of this touches a model, it is fully reproducible: run this block again and you get these identical numbers.

Stage 2: Run Docent Live#

Now we need real predictions to score. We drop in the canonical Docent — retrieval over the Meridian docs plus one claude-haiku-4-5 call — and ask it every golden question. Docent reads its API key from the environment (anthropic.Anthropic() with no arguments); we never hard-code a key. Keep the run small: eight short questions, max_tokens=200, one call each.

import anthropic
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

DOCS = [
    {"id": "ratelimits", "title": "Rate Limits",
     "text": "The Free plan allows 60 requests per minute. The Pro plan allows 600 requests per minute. Exceeding the "
             "limit returns HTTP 429 with a Retry-After header giving the seconds to wait. Rate limits are applied per "
             "API key, not per account."},
    {"id": "plans", "title": "Plans and Pricing",
     "text": "Meridian has three plans. Free costs 0 dollars per month and includes 1 GB of storage. Pro costs 25 dollars "
             "per month and includes 50 GB of storage. Scale costs 199 dollars per month and includes 500 GB of storage. "
             "All plans include unlimited read replicas."},
    {"id": "errors", "title": "Error Codes",
     "text": "Meridian uses standard HTTP status codes. 400 means a malformed request, 401 means a missing or invalid API "
             "key, 403 means the key is valid but lacks permission, 404 means the resource does not exist, and 429 means "
             "the rate limit was exceeded. 5xx codes indicate a server-side error and should be retried with backoff."},
    {"id": "backups", "title": "Backups",
     "text": "Automatic daily backups are retained for 7 days on the Free plan, 30 days on Pro, and 90 days on Scale. "
             "Backups run at 03:00 UTC. Point-in-time recovery to any second in the retention window is available on the "
             "Scale plan. Restoring a backup creates a new database and never overwrites the original."},
    {"id": "sdks", "title": "SDKs",
     "text": "Official SDKs are available for Python, JavaScript, and Go. Install the Python SDK with 'pip install meridian'. "
             "The SDK reads the MERIDIAN_API_KEY environment variable by default. All SDKs retry idempotent requests up to "
             "3 times on 5xx errors with exponential backoff."},
    {"id": "deletion", "title": "Deleting Data",
     "text": "Deleting a database is immediate and irreversible once the 24-hour grace period ends. During the grace period "
             "the database is soft-deleted and can be restored from the dashboard. After 24 hours the data and its backups "
             "are permanently purged. Deletion requires a live-mode key with admin permission."},
]

def retrieve(question, docs, k=2):
    q = set(question.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())), reverse=True)
    return scored[:k]

def docent_answer(question, docs, model="claude-haiku-4-5"):
    ctx = "\n\n".join(f"[{d['id']}] {d['title']}\n{d['text']}" for d in retrieve(question, docs))
    prompt = (f"Answer the question using ONLY the context. If the context does not contain the answer, say "
              f"\"I don't have that in the docs.\" Be concise.\n\nContext:\n{ctx}\n\nQuestion: {question}")
    msg = client.messages.create(model=model, max_tokens=200,
                                 messages=[{"role": "user", "content": prompt}])
    return msg.content[0].text

# collect one real answer per golden question
answers = [docent_answer(item["q"], DOCS) for item in GOLDEN]

print("A couple of real Docent answers:\n")
print("Q:", GOLDEN[1]["q"])
print("A:", answers[1], "\n")
print("Q:", GOLDEN[5]["q"])
print("A:", answers[5])

A couple of real Docent answers:

Q: How many requests per minute does the Pro plan allow?
A: The Pro plan allows 600 requests per minute.

Q: What does a 401 error mean?
A: A 401 error means a missing or invalid API key.

Both answers are factually correct — Docent pulled the right numbers from the right pages. But look at the shape: it never replies with the bare reference. It answers in a full, helpful sentence (“The Pro plan allows 600 requests per minute.”), because that is what a good documentation assistant does. That habit is harmless to a reader and, as you’re about to see, catastrophic for exact match. These are one real run’s answers; because the model samples, a re-run might phrase the 401 answer as “It means the API key is missing or invalid” or answer the rate-limit code as bare “HTTP 429.” — close, but not byte-identical. That variation is exactly why we score, and it’s why the scorer must be the deterministic half of the system.

Stage 3: Score Every Answer on Every Metric#

With eight real answers and three scorers, the scorecard is a nested loop: for each (answer, reference) pair, compute all three metrics into a per-item row, then average each column to get the aggregate. The per-item table shows you where Docent wins and loses; the aggregates give you the single-number summary you’d track over time.

def build_scorecard(golden, answers):
    rows = []
    for item, ans in zip(golden, answers):
        rows.append(dict(
            ref=item["ref"],
            em=exact_match(ans, item["ref"]),
            f1=f1_score(ans, item["ref"]),
            rl=rouge_l(ans, item["ref"]),
        ))
    n = len(rows)
    agg = dict(
        em=sum(r["em"] for r in rows) / n,
        f1=sum(r["f1"] for r in rows) / n,
        rl=sum(r["rl"] for r in rows) / n,
    )
    return rows, agg

rows, agg = build_scorecard(GOLDEN, answers)

print(f"{'#':>2} {'reference':<26} {'EM':>3} {'F1':>6} {'ROUGE-L':>8}")
for i, r in enumerate(rows, 1):
    print(f"{i:>2} {r['ref'][:25]:<26} {r['em']:>3} {r['f1']:>6.2f} {r['rl']:>8.2f}")
print("-" * 48)
print(f"   {'MEAN':<26} {agg['em']:>3.2f} {agg['f1']:>6.2f} {agg['rl']:>8.2f}")

 # reference                   EM     F1  ROUGE-L
 1 429                          0   0.17     0.15
 2 600 requests per minute      0   0.73     0.67
 3 25 dollars per month         0   0.60     0.55
 4 90 days                      0   0.36     0.33
 5 MERIDIAN_API_KEY             0   0.22     0.40
 6 A missing or invalid API     0   0.77     0.75
 7 24 hours                     0   0.24     0.21
 8 Only the Scale plan          0   0.43     0.32
------------------------------------------------
   MEAN                       0.00   0.44     0.42

There it is — the real scorecard for one live run. Every one of Docent’s answers is correct, and yet mean exact match is 0.00: not a single answer matched its reference verbatim after normalization, because Docent always wraps the fact in a sentence. F1 and ROUGE-L are far more forgiving, landing at 0.44 and 0.42, because they credit the reference words that appear inside the longer answer. The short references pay the steepest F1 penalty — “429” as a lone token gets swamped by the ten words around it in “The rate limit returns HTTP 429…”, so item 1 scores just 0.17 despite being perfectly right. Aggregates will drift a few points on every re-run as the model rephrases, but the story is stable: EM near zero, F1 and ROUGE-L in the middle, tracking each other closely.

One subtlety worth stating out loud: the scorer is deterministic even though the model is not. If you freeze the eight answers in a list and re-run build_scorecard, you get byte-identical numbers every time — we verified that. What moves between runs is only Docent’s phrasing. That split is the healthy design: the thing you’re measuring (the model) is allowed to vary, but the ruler you measure it with (the metrics) must not.

Stage 4: Read the Scorecard#

A scorecard is only worth building if you can read it, and reading it means explaining why the columns disagree rather than picking a favorite. Here is what this one is telling us.

Exact match is strict to the point of uselessness here. It demands the normalized answer equal the normalized reference character for character. Docent is a helpful assistant that answers in sentences, so EM is 0.00 across the board even though every answer is right. EM earns its keep only when the reference is a closed, canonical token the model can reproduce exactly — a status code returned alone, a boolean, a single enum value. For free-form answers it measures formatting compliance, not correctness, and reports mostly noise.

F1 gives partial credit for the right words, regardless of order. It treats each answer as a bag of tokens and rewards overlap with the reference, which is why it climbs to 0.44: “The Pro plan allows 600 requests per minute” shares every reference word, so item 2 scores 0.73. F1’s blind spot is length — a correct short reference buried in a long answer scores low on precision (item 1’s “429”), and F1 can’t tell a correct extra clause from a wrong one, since both just dilute the overlap.

ROUGE-L rewards the right words in the right order. By scoring the longest common subsequence it’s stricter than F1 about sequence, which is why it usually sits just below F1 (0.42 vs 0.44) and drops harder when the model reorders a phrase — recall the Stage 1 row where “missing or invalid API key” reordered from the reference and ROUGE-L fell to 0.40 while F1 held at 0.77. ROUGE-L was designed for summarization, where phrase order carries meaning; for a short factual answer it mostly adds a mild word-order penalty on top of what F1 already saw.

So which do you trust for Docent’s short factual answers? Lean on F1, and read EM as a separate signal that’s only meaningful for the handful of genuinely canonical references. ROUGE-L rarely tells you anything F1 didn’t here — it comes into its own for longer generated text, not one-line facts. But the deepest lesson is where all three fail together. Every one of these metrics scores surface overlap, not correctness. An answer that paraphrases the reference perfectly — “It costs twenty-five dollars a month” versus “25 dollars per month” — shares almost no tokens and is punished by all three, even though it’s flawless. And the mirror-image failure is worse: an answer that copies the reference’s words but states them wrongly (“the Free plan costs 25 dollars per month”) can score high overlap while being flatly false. Word overlap is a proxy for correctness, and a leaky one.

That gap is the entire motivation for the next module. When correctness and surface form come apart — paraphrases, reorderings, subtle factual errors dressed in the right vocabulary — a reference-based metric can’t help you, because it never understood the answer in the first place. To score meaning, you need a judge that reads. That’s LLM-as-judge, and it’s where Module 4 begins.

Practice Exercises#

Exercise 1: Add a best_of column#

Right now each metric scores an answer against a single reference. But many questions have several acceptable phrasings — “24 hours”, “one day”, “a 24-hour grace period”. Extend the scorer so each golden item can carry a list of references, and score EM/F1/ROUGE-L as the maximum over that list. Re-run the scorecard and see how much mean F1 rises once “correct but differently worded” stops being penalized.

Exercise 2: Score the same fixed answers twice to prove reproducibility#

Capture the eight live answers into a plain Python list, then run build_scorecard on that fixed list twice and assert the two aggregate dicts are equal. This separates the deterministic scorer from the stochastic model and gives you a regression test you could drop into CI.

Exercise 3: Flag disagreement between metrics#

The interesting rows are where the metrics disagree — high F1 but low ROUGE-L usually means the model reordered the reference, and any answer where EM is 0 but F1 is above 0.8 is “correct but verbose”. Add a flag column that prints a short label for each item based on the gap between its metrics, and print only the flagged rows.

Summary#

You built a working reference-metric scorecard for Docent end to end. You assembled the pieces — an eight-question golden set and three scorers (normalized exact match, token F1, and ROUGE-L via rouge-score) all sharing one SQuAD-style normalizer, so every column scores the same cleaned text. You ran Docent live over the golden set and saw it answer correctly but always in full sentences. You scored every answer on every metric into a per-item table and aggregated it to a real scorecard — mean EM 0.00, F1 0.44, ROUGE-L 0.42 for this run — while confirming the scorer itself is deterministic even though the model isn’t. And you read the scorecard, explaining why EM is too strict for a sentence-writing assistant, why F1 is the most useful column for short factual answers, why ROUGE-L mostly echoes F1 with a word-order penalty, and where all three fall short — on paraphrases and on wrong answers wearing the right words.

Key Concepts#

• Scorecard — a per-item table (question, EM, F1, ROUGE-L) plus column aggregates that shows multiple metrics side by side, so their disagreement becomes a signal rather than a single number hiding the story.
• Shared normalization — every metric calls the same SQuAD-style normalize, making the columns genuinely comparable; the only thing that varies down a row is each metric’s definition of “close”.
• Deterministic scorer, stochastic model — the metric code returns byte-identical numbers on the same answers every run, while the model rephrases; the ruler must not move even when the thing measured does.
• EM vs F1 vs ROUGE-L — EM is all-or-nothing (trustworthy only for canonical tokens), F1 gives order-free partial credit (best for short factual answers), ROUGE-L adds a word-order penalty (built for longer text).
• Overlap is not correctness — all three metrics score surface overlap, so they punish correct paraphrases and can reward fluent-but-false answers; that gap is what LLM-as-judge exists to close.

Why This Matters#

A reference-metric scorecard is the cheapest, fastest, most reproducible eval you can run — no model in the loop, no human labeling, milliseconds per item, and the same number every time — which makes it the natural first gate in CI and the baseline every fancier method is measured against. But this project also taught you its ceiling. Because these metrics only ever compare strings, they can tell you an answer looks like the reference but never that it’s true; a scorecard of confident overlap numbers can quietly bless a paraphrase as wrong or a falsehood as right. Knowing exactly where that ceiling is — trusting F1 for short factual answers, distrusting all three on paraphrase and correctness — is what lets you use deterministic metrics for what they’re good at and reach for a smarter judge for everything else.

Continue Building Your Skills#

You just turned a module’s worth of individual metrics into one artifact that a team can actually run: a scorecard that takes Docent’s real answers and reports exact match, token F1, and ROUGE-L side by side, with aggregates you can track over time and a deterministic scorer you can drop into CI. More importantly, you learned to read it — to treat the disagreement between columns as the signal, to trust F1 for short factual answers, and to see clearly where every reference metric hits its ceiling. That ceiling is where the course goes next. In Module 4 you’ll bring a model into the loop as a judge, so you can finally score the paraphrases, reorderings, and dressed-up falsehoods that string overlap can never catch.