Lesson 1 - Exact Match & Normalization

Welcome to Exact Match & Normalization#

In Module 2 you built and audited a golden set for Docent – our assistant for the Meridian serverless database docs – so now every question comes paired with a known-correct reference answer. This module turns those labels into scores you can compute with plain code: no model in the loop, no human grading, the same number every run. We start with the simplest metric of all, the one every other reference metric is measured against: exact match.

Exact match asks one question – does the model’s output equal the reference? – and answers it with ==. That’s the whole metric. It’s perfectly reproducible, costs nothing, and needs no judgement. But raw string equality is also brutally literal: it thinks "HTTP 429." and "429" are two different answers, even though any Meridian user would call them the same. The fix isn’t to abandon exact match; it’s to normalize both strings first – lowercase them, strip punctuation, drop articles, tidy whitespace – so the comparison judges the fact, not the formatting. That single idea takes exact match from useless to genuinely useful for short factoids.

This lesson implements both pieces for real, scores a table of realistic Docent answers, and shows you the exact moment normalization rescues a pile of correct answers a raw comparison had thrown away.

In this lesson, you will:

• Implement exact_match(pred, ref) and the classic SQuAD-style normalize_answer(s) from scratch
• Score a realistic set of Docent predictions against the canonical golden references and watch the aggregate jump from raw 0.250 to normalized 0.875
• Learn when exact match is the right metric (short, closed-form factoids) and when it’s the wrong one (any answer with legitimate phrasing variation)
• See exact match’s hard limit – no partial credit – that motivates token-level F1 in the next lesson

Exact match: the metric is ==#

Exact match (EM) is the strictest, cheapest reference metric. Given a model prediction pred and a golden reference ref, it scores 1 if the two strings are identical and 0 otherwise:

def exact_match(pred, ref):
    return int(pred == ref)

print(exact_match("429", "429"))
print(exact_match("HTTP 429.", "429"))
print(exact_match("429.", "429"))

1
0
0

The first case is a clean hit. The other two are the problem. Docent answered the rate-limit question with "HTTP 429." – which is correct, and arguably more helpful than the bare reference – and EM scored it 0, the same score it would give the answer "potato". "429." with a trailing period fares no better. Raw exact match has no notion of “close”: "HTTP 429" != "429" != "429.", even though all three name the same status code.

This is not a bug you can shrug off. On real model output, trivial formatting differences – a trailing period, a capital letter, the word “the” – are everywhere, and each one silently turns a correct answer into a zero. Left uncorrected, raw EM doesn’t measure whether Docent is right; it measures whether Docent is right and happens to match your reference’s exact punctuation. Those are very different questions, and only the first one matters.

Normalization: judge the fact, not the formatting#

The cure for over-strict EM is to run both strings through the same normalization function before comparing them, collapsing away the differences that don’t change meaning. The canonical version comes from the SQuAD reading-comprehension benchmark, and it does four things, in order:

1. Lowercase – so "MERIDIAN" and "meridian" match.
2. Remove punctuation – so a trailing period or a stray comma doesn’t count.
3. Drop the articles a, an, the – so "the Scale plan" and "Scale plan" match.
4. Collapse whitespace – strip the ends and squeeze internal runs of spaces to one, so "90 days" and " 90 days " match.

Here it is, and it’s short enough to read in one sitting:

import re
import string

def normalize_answer(s):
    """SQuAD-style normalization: lowercase, drop punctuation, drop articles, fix whitespace."""
    def lower(text):
        return text.lower()

    def remove_punc(text):
        return "".join(ch for ch in text if ch not in set(string.punctuation))

    def remove_articles(text):
        return re.sub(r"\b(a|an|the)\b", " ", text)

    def white_space_fix(text):
        return " ".join(text.split())

    return white_space_fix(remove_articles(remove_punc(lower(s))))

The order matters. Lowercasing first means the article regex only has to match lowercase the, not The or THE. Removing punctuation before collapsing whitespace means a stripped period leaves behind a space that the final step tidies up. Watch every stage fire on one messy-but-correct prediction of the 401 answer, and see it land exactly on the normalized reference:

pred = "  The missing, or invalid API key.  "
ref = "A missing or invalid API key"

s1 = pred.lower()
s2 = "".join(ch for ch in s1 if ch not in set(string.punctuation))
s3 = re.sub(r"\b(a|an|the)\b", " ", s2)
s4 = " ".join(s3.split())

print(f"0. raw .............. {pred!r}")
print(f"1. lowercase ........ {s1!r}")
print(f"2. remove punct ..... {s2!r}")
print(f"3. drop articles .... {s3!r}")
print(f"4. fix whitespace ... {s4!r}")
print()
print(f"normalized reference : {normalize_answer(ref)!r}")
print(f"normalized exact match: {int(s4 == normalize_answer(ref))}")

0. raw .............. '  The missing, or invalid API key.  '
1. lowercase ........ '  the missing, or invalid api key.  '
2. remove punct ..... '  the missing or invalid api key  '
3. drop articles .... '    missing or invalid api key  '
4. fix whitespace ... 'missing or invalid api key'

normalized reference : 'missing or invalid api key'
normalized exact match: 1

The prediction and the reference differed by capitalization, a comma, a period, the word “the”, and a pile of stray spaces – five differences, none of them a difference in meaning – and normalization erased all five, leaving two identical strings. That’s the whole trick: normalization doesn’t make EM lenient, it makes it fair, by refusing to let cosmetic formatting masquerade as a wrong answer.

Scoring Docent: raw EM versus normalized EM#

Now put it to work on a realistic batch. Below are eight Docent predictions paired with their golden references drawn from the canonical set. Two answers are already byte-perfect; five more are correct but formatted differently (a trailing period, mixed case, extra spaces, a dropped article); and the last is genuinely wrong – Docent claimed point-in-time recovery is on “All paid plans” when the docs say it’s Scale only. We score every pair two ways – raw EM and normalized EM – and aggregate:

def normalized_exact_match(pred, ref):
    return int(normalize_answer(pred) == normalize_answer(ref))

# (question, Docent prediction, golden reference)
PAIRS = [
    ("What HTTP status code does Meridian return when you exceed the rate limit?",
     "429", "429"),
    ("Which environment variable does the Python SDK read for the API key?",
     "MERIDIAN_API_KEY", "MERIDIAN_API_KEY"),
    ("How much does the Pro plan cost per month?",
     "25 dollars per month.", "25 dollars per month"),
    ("How many requests per minute does the Pro plan allow?",
     "600 Requests Per Minute", "600 requests per minute"),
    ("How long are backups retained on the Scale plan?",
     "  90 days  ", "90 days"),
    ("What does a 401 error mean?",
     "Missing or invalid API key", "A missing or invalid API key"),
    ("How long is the grace period before a deleted database is purged?",
     "24  hours", "24 hours"),
    ("Which plans support point-in-time recovery?",
     "All paid plans", "Only the Scale plan"),
]

print(f"{'reference':<28} {'prediction':<26} {'raw':>4} {'norm':>5}")
print("-" * 66)
raw_hits, norm_hits = 0, 0
for q, pred, ref in PAIRS:
    raw = exact_match(pred, ref)
    norm = normalized_exact_match(pred, ref)
    raw_hits += raw
    norm_hits += norm
    print(f"{ref:<28} {repr(pred):<26} {raw:>4} {norm:>5}")

n = len(PAIRS)
print("-" * 66)
print(f"raw exact match ........ {raw_hits}/{n} = {raw_hits / n:.3f}")
print(f"normalized exact match . {norm_hits}/{n} = {norm_hits / n:.3f}")

reference                    prediction                  raw  norm
------------------------------------------------------------------
429                          '429'                         1     1
MERIDIAN_API_KEY             'MERIDIAN_API_KEY'            1     1
25 dollars per month         '25 dollars per month.'       0     1
600 requests per minute      '600 Requests Per Minute'     0     1
90 days                      '  90 days  '                 0     1
A missing or invalid API key 'Missing or invalid API key'    0     1
24 hours                     '24  hours'                   0     1
Only the Scale plan          'All paid plans'              0     0
------------------------------------------------------------------
raw exact match ........ 2/8 = 0.250
normalized exact match . 7/8 = 0.875

Read the two columns. Raw EM scores 2 of 8 (0.250) – only the two byte-perfect answers survive, and Docent looks broken. Normalized EM scores 7 of 8 (0.875) – the five correctly-answered-but-differently-formatted questions all flip to 1, and the number now reflects reality: Docent got seven of these eight facts right. The gap between 0.250 and 0.875 is entirely formatting noise that raw EM mistook for wrongness. This is deterministic arithmetic over fixed strings, so it’s fully reproducible – run it again and the numbers are identical:

print(f"run 2 -> raw {raw_hits}/{n} = {raw_hits / n:.3f}"
      f" | normalized {norm_hits}/{n} = {norm_hits / n:.3f}")

run 2 -> raw 2/8 = 0.250 | normalized 7/8 = 0.875

Notice which pair normalization did not rescue: the last one. "All paid plans" normalizes to all paid plans and "Only the Scale plan" normalizes to only scale plan – still different, because they are different, and Docent is simply wrong. That’s the behavior you want: normalization forgives formatting and only formatting. It should never turn a wrong answer into a right one.

When exact match is the right metric – and when it lies#

Even normalized, EM is only trustworthy for a specific shape of question: short, closed-form factoids with essentially one correct surface answer. Docent’s golden set is full of these, and they’re exactly where EM shines:

• "What HTTP status code does Meridian return when you exceed the rate limit?" → "429"
• "How many requests per minute does the Pro plan allow?" → "600 requests per minute"
• "Which environment variable does the Python SDK read for the API key?" → "MERIDIAN_API_KEY"
• "How long is the grace period before a deleted database is purged?" → "24 hours"

For answers like these, there’s a canonical string (a status code, an env var name, a fixed number) and any faithful answer reduces to it after normalization. EM is the metric here: it’s exact, free, reproducible, and correct. Reaching for an LLM judge to grade "is 429 equal to 429?" would be slower, costlier, and less reliable than a ==.

EM starts lying the moment an answer has legitimate phrasing variation. Take the region question, whose reference is a full sentence:

ref = "No; the region is fixed at creation and you must export and re-import into a new database to move regions."
pred = "No, you can't change a region after creation -- you'd have to export the data and re-import it into a new database."

print("normalized EM:", normalized_exact_match(pred, ref))

normalized EM: 0

That prediction is completely correct – same facts, same conclusion – and normalized EM still scores it 0, because the two sentences share meaning but not wording, and normalization only smooths formatting, not paraphrase. Push this far enough and EM becomes actively misleading: a nine-word answer that differs from the reference by a single word scores 0, the exact same score as total nonsense. EM gives no partial credit. It’s all-or-nothing, so it can’t distinguish “almost perfect” from “completely wrong,” and for any answer longer than a few tokens that’s a fatal blind spot. That limitation is the entire motivation for the next lesson’s metric, token-level F1, which measures word overlap and hands out the partial credit EM refuses to.

A live Docent answer, normalized and scored#

Everything above used fixed strings so the numbers reproduce exactly. To see why normalization matters on real model output, here’s one small live call to Docent for the rate-limit question, then the same scoring pipeline applied to whatever it says. (Model output varies run to run, so your exact string may differ from the one recorded here – but the lesson it teaches is stable.)

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

# ... DOCS, retrieve(), and docent_answer() are the canonical Docent from Module 1 ...

q = "What HTTP status code does Meridian return when you exceed the rate limit?"
ref = "429"
pred = docent_answer(q, DOCS)

print("raw answer .......", repr(pred))
print("normalized .......", repr(normalize_answer(pred)))
print("raw EM vs '429' ..", exact_match(pred, ref))
print("norm EM vs '429' .", normalized_exact_match(pred, ref))

raw answer ....... 'HTTP 429.'
normalized ....... 'http 429'
raw EM vs '429' .. 0
norm EM vs '429' . 0

This real run is a perfect cautionary tale. Docent answered "HTTP 429." – correct, helpful, and it still scores 0 under both raw and normalized EM. Why does normalization fail here when it rescued five answers earlier? Because normalization only removes formatting – case, punctuation, articles, whitespace. It does not remove extra content: the token http is a real word the model added, and no normalization rule strips it, so "http 429" will never equal "429". That’s the honest edge of this metric. Normalization makes EM fair to formatting differences, but a model that answers with even one extra meaningful word still gets zero credit – which, again, is exactly the gap token-level F1 exists to close. You could patch this specific case by loosening the reference or asking Docent to reply with only the code, but the general fix is a metric that scores overlap instead of demanding identity.

Practice Exercises#

Exercise 1: Break normalization on purpose#

Find a Docent prediction that is genuinely correct for the "How much does the Pro plan cost per month?" question (reference "25 dollars per month") but that normalized exact match still scores 0. Then explain in one sentence which category of difference – formatting or content – your example exploits.

Exercise 2: Make normalization number-aware#

The reference "600 requests per minute" and a prediction of "600 rpm" are the same fact, but even normalized EM scores them 0. Write a small expand_units helper that maps common Meridian abbreviations ("rpm" → "requests per minute", "gb" → "gigabytes") and apply it inside a custom normalizer, then confirm "600 rpm" now matches. What new failure could an over-eager expansion introduce?

Exercise 3: Route by answer shape#

Given the eight golden references in this lesson, decide for each whether exact match is a trustworthy metric or a misleading one, based only on the reference’s shape. Group them into two lists and justify the split in a sentence. Which references would you re-route to token-level F1 in the next lesson?

Summary#

Exact match is the simplest reference metric there is – int(pred == ref) – and its strengths are exactly the strengths of deterministic metrics in general: zero cost, zero randomness, perfectly reproducible, no model or human in the loop. Its weakness is that raw string equality is brutally strict, treating "HTTP 429.", "429.", and "429" as three different answers and scoring a correct-but-differently-formatted answer the same 0 as total nonsense. The fix is normalization: run both strings through the SQuAD-style normalize_answer – lowercase, strip punctuation, drop articles, collapse whitespace – before comparing, so the metric judges the fact and not the formatting. On eight realistic Docent pairs that lifted the aggregate from a raw 2/8 = 0.250 to a normalized 7/8 = 0.875, flipping five correct answers from 0 to 1 while correctly leaving the one genuinely wrong answer at 0. Exact match is the right metric for short, closed-form factoids (a status code, an env var, a fixed number) and the wrong one for anything with legitimate phrasing variation, because it gives no partial credit – a limitation a live Docent answer of "HTTP 429." made concrete, and one that token-level F1 exists to fix.

Key Concepts#

• Exact match (EM) – int(pred == ref); scores 1 only if the prediction string equals the reference exactly. Free, deterministic, reproducible, and brutally strict.
• Normalization – the SQuAD-style normalize_answer: lowercase, remove punctuation, drop the articles a/an/the, collapse whitespace. Applied to both strings before comparing so cosmetic formatting stops counting as a wrong answer.
• Raw vs normalized EM – on the same eight Docent pairs, raw EM = 0.250 and normalized EM = 0.875; the gap is pure formatting noise that raw EM mistook for wrongness.
• When EM fits – short, closed-form factoids with one canonical surface answer ("429", "MERIDIAN_API_KEY", "24 hours"); when it lies – any answer with legitimate paraphrase, like the full-sentence region answer.
• No partial credit – EM is all-or-nothing, so a nine-word answer off by one word scores the same 0 as nonsense, and even normalization can’t rescue an answer with one extra meaningful token ("http 429" vs "429"). This motivates token-level F1.

Why This Matters#

Exact match is the metric teams reach for first because it’s trivial to implement – and the one they most often misuse, by running it over answers that have many correct phrasings and then concluding their model is terrible when it’s really just paraphrasing. Understanding both halves – that normalization is mandatory to make EM fair, and that even normalized EM only fits short closed-form answers – is what lets you use it correctly: as a fast, free, reproducible gate on the factoid questions where it’s genuinely the best tool, and nowhere else. Every other metric in this module and this course is, in some sense, a response to exact match’s rigidity, so getting a clear-eyed feel for what it does and doesn’t measure is the foundation the rest of your reference-metric toolkit is built on. Next you’ll add the partial credit EM refuses to give.

Continue Building Your Skills#

You now have the two smallest, sturdiest tools in the reference-metric kit: exact_match, which turns “is this answer right?” into a ==, and normalize_answer, which makes that == fair by judging the fact instead of the formatting. You watched them lift Docent’s score from a misleading 0.250 to an honest 0.875, and you saw precisely where they stop – the moment an answer carries legitimate paraphrase or one extra meaningful word, exact match hands out a zero it doesn’t deserve. That all-or-nothing rigidity is the gap the next lesson closes: token-level F1 scores the overlap between a prediction and a reference, so a Docent answer that gets most of a multi-word fact right finally earns most of the credit – the partial-credit metric that makes reference scoring usable beyond one-token factoids.