Lesson 2 - Token-Level F1 & Overlap Metrics

Welcome to Token-Level F1 & Overlap Metrics#

In Lesson 1 you scored Docent – our assistant for the Meridian serverless database docs – with exact match: normalize both strings, then check whether they’re identical. That gave you a fast, reproducible number, but it also gave you a harsh one. Exact match is a switch, not a dial. Docent can answer "The rate limit returns HTTP status code 429" when the reference is "429", get the fact exactly right, and still score a flat 0, because the strings aren’t identical. A completely wrong answer scores 0 too. Exact match cannot tell those two apart, and for question answering that’s a real problem: most answers are neither perfect strings nor total nonsense – they’re mostly right, and you want a metric that says so.

This lesson introduces the metric that gives partial credit: token-level F1, the same F1 the SQuAD reading-comprehension benchmark uses. Instead of comparing two strings character-for-character, it compares two bags of tokens and measures how much they overlap. A mostly-correct answer earns a score between 0 and 1 instead of a binary miss, so Docent’s near-misses stop looking identical to its disasters.

In this lesson, you will:

• Compute precision, recall, and their harmonic mean over normalized tokens to get a token-level F1 score
• See why F1 beats exact match for QA: verbose answers lose precision, terse answers lose recall, and mostly-right answers land in between instead of at zero
• Build a real, deterministic f1_score function and score four Docent answers, contrasting mean EM (0.25) against mean F1 (0.51) on the same set
• Meet F1’s blind spots – it ignores word order and meaning – which is exactly what motivates the semantic and judge metrics later in the course

From strings to bags of tokens#

Exact match asks one question: are these two strings identical after normalization? Token-level F1 asks a softer one: how many words do these two answers share? To answer it, you stop treating each answer as a single string and start treating it as a bag of tokens – the multiset of words you get after normalizing and splitting on whitespace.

We reuse the exact same normalization from Lesson 1, and that’s deliberate: F1 and EM should agree on what counts as “the same word.” Lowercase the text, strip punctuation, drop the articles a, an, and the, and collapse whitespace. So the answer "Only the Scale plan" becomes the token bag ["only", "scale", "plan"] – the article the is gone, so it never inflates or deflates the overlap.

Once both the prediction and the reference are bags of tokens, one number drives everything: the count of tokens they share. From that single count you get two views of quality, which is the whole idea behind F1.

Precision, recall, and the harmonic mean#

Let $\text{pred}$ be the bag of prediction tokens, $\text{ref}$ the bag of reference tokens, and $|\text{pred} \cap \text{ref}|$ the number of tokens they share. Precision asks what fraction of the words Docent said were also in the reference? – it punishes padding. Recall asks what fraction of the reference words did Docent actually say? – it punishes omission.

A verbose answer has a large $|\text{pred}|$, so even if it contains every reference word (recall = 1.0), its precision is low – lots of extra words dilute it. A terse answer that drops half the reference has high precision (everything it said was right) but low recall (it left facts out). You want a single number that’s only high when both are high, and that’s the harmonic mean – the F1 score:

The harmonic mean is the right average here because it’s dominated by the smaller of the two values. If precision is 1.0 but recall is 0.1, the arithmetic mean would be a generous 0.55, but the harmonic mean is about 0.18 – it refuses to let one strong half hide a weak one. An answer only earns a high F1 by being both on-topic (precise) and complete (high recall). If either precision or recall is 0, F1 is 0.

Building a real f1_score#

Here is a deterministic implementation – no model, no randomness, just arithmetic over tokens, so it returns identical numbers every run. normalize is the Lesson 1 normalization (lowercase, strip punctuation, drop articles, collapse whitespace), and f1_score returns the precision, recall, and F1 for a prediction against a reference. We use collections.Counter so that repeated words are matched as a multiset – the intersection Counter(pred) & Counter(ref) keeps the minimum count of each shared token, which is the standard SQuAD definition.

import re, string
from collections import Counter

def normalize(text):
    text = text.lower()
    text = "".join(ch for ch in text if ch not in set(string.punctuation))
    text = re.sub(r"\b(a|an|the)\b", " ", text)      # drop articles
    return " ".join(text.split())                     # collapse whitespace

def f1_score(pred, ref):
    pred_toks = normalize(pred).split()
    ref_toks = normalize(ref).split()
    if not pred_toks and not ref_toks:                # both empty -> perfect
        return 1.0, 1.0, 1.0
    if not pred_toks or not ref_toks:                 # exactly one empty -> zero
        return 0.0, 0.0, 0.0
    shared = sum((Counter(pred_toks) & Counter(ref_toks)).values())
    if shared == 0:
        return 0.0, 0.0, 0.0
    precision = shared / len(pred_toks)
    recall = shared / len(ref_toks)
    f1 = 2 * precision * recall / (precision + recall)
    return precision, recall, f1

CASES = [
    ("exact hit",            "429",                                          "429"),
    ("verbose but correct",  "The rate limit returns HTTP status code 429.", "429"),
    ("terse partial",        "Only Scale",                                   "Only the Scale plan"),
    ("wrong answer",         "404",                                          "429"),
]

print("Per-answer precision / recall / F1")
print("-" * 66)
print(f"{'case':<22}{'P':>8}{'R':>8}{'F1':>8}   pred vs ref")
for name, pred, ref in CASES:
    p, r, f = f1_score(pred, ref)
    print(f"{name:<22}{p:>8.2f}{r:>8.2f}{f:>8.2f}   {pred!r} | {ref!r}")

Per-answer precision / recall / F1
------------------------------------------------------------------
case                         P       R      F1   pred vs ref
exact hit                 1.00    1.00    1.00   '429' | '429'
verbose but correct       0.14    1.00    0.25   'The rate limit returns HTTP status code 429.' | '429'
terse partial             1.00    0.67    0.80   'Only Scale' | 'Only the Scale plan'
wrong answer              0.00    0.00    0.00   '404' | '429'

Read the four rows and F1’s behavior is exactly what the formulas promised:

• Exact hit ("429" vs "429"): every token shared, so precision, recall, and F1 are all 1.0 – F1 agrees with exact match when the answer is perfect.
• Verbose but correct ("The rate limit returns HTTP status code 429." vs "429"): recall is a perfect 1.0 because the one reference token 429 is present, but precision is just 0.14 – Docent said seven tokens and only one was the reference. The right fact is buried in padding, and F1 (0.25) reflects that it’s correct-but-bloated, not perfect.
• Terse partial ("Only Scale" vs "Only the Scale plan"): precision is 1.0 (both words Docent said are in the reference) but recall is 0.67 because it dropped plan. F1 is 0.80 – a strong partial credit for a mostly-complete answer.
• Wrong answer ("404" vs "429"): zero shared tokens, so F1 is 0.0 – F1 still gives a genuine miss the zero it deserves.

The payoff: partial credit that exact match cannot give#

Now run both metrics over the same four answers and watch the aggregate diverge. Exact match returns 1 only for the perfect string; F1 rewards the two near-correct answers that EM threw away.

def exact_match(pred, ref):
    return 1 if normalize(pred) == normalize(ref) else 0

em_scores = [exact_match(p, r) for _, p, r in CASES]
f1_scores = [f1_score(p, r)[2] for _, p, r in CASES]

print("EM vs F1 on the same set")
print("-" * 66)
for (name, _, _), em, f in zip(CASES, em_scores, f1_scores):
    print(f"{name:<22} EM={em}   F1={f:.2f}")

mean_em = sum(em_scores) / len(em_scores)
mean_f1 = sum(f1_scores) / len(f1_scores)
print(f"{'aggregate':<22} mean EM={mean_em:.2f}   mean F1={mean_f1:.2f}")

EM vs F1 on the same set
------------------------------------------------------------------
exact hit              EM=1   F1=1.00
verbose but correct    EM=0   F1=0.25
terse partial          EM=0   F1=0.80
wrong answer           EM=0   F1=0.00
aggregate              mean EM=0.25   mean F1=0.51

The aggregate tells the story. Exact match scores this set 0.25 – only one of the four answers was a perfect string. But three of the four answers are not worthless: two of them are correct or nearly so, and one is genuinely wrong. F1 scores the set 0.51, twice as high, because it credits the verbose answer (0.25) and the terse one (0.80) for the facts they got right while still zeroing the wrong answer. That 0.25-versus-0.51 gap is the difference between a metric that only recognizes perfection and one that measures how close Docent got. For question answering, where most answers are shades of “mostly right,” that gap is why the SQuAD benchmark and most QA leaderboards report F1 alongside (or instead of) exact match.

Because it’s pure arithmetic, the whole thing is reproducible – run it a second time and every number is identical:

em2 = [exact_match(p, r) for _, p, r in CASES]
f12 = [round(f1_score(p, r)[2], 2) for _, p, r in CASES]
print("run 2 -> EM:", em2, "| F1:", f12,
      "| mean EM:", round(sum(em2) / len(em2), 2),
      "| mean F1:", round(sum(f12) / len(f12), 2))

run 2 -> EM: [1, 0, 0, 0] | F1: [1.0, 0.25, 0.8, 0.0] | mean EM: 0.25 | mean F1: 0.51

Same EM, same per-answer F1, same aggregate – a deterministic metric earns the trust that a model-in-the-loop score has to fight for.

Where token F1 quietly lies#

F1 is a real improvement, but it buys partial credit with a strong assumption: that an answer is nothing but a bag of words. Throwing away sentence structure is what makes the metric cheap and reproducible – and it’s also where it breaks. Two failure modes matter for Docent.

It ignores word order. A bag of tokens has no notion of sequence, so two answers with opposite meanings but identical word sets score the same. Consider a question about error codes where the reference is "429 not 200". Docent could answer "200 not 429" – the exact opposite claim – and F1 cannot tell the difference:

ref = "429 not 200"
for pred in ["429 not 200", "200 not 429"]:
    p, r, f = f1_score(pred, ref)
    print(f"pred={pred!r:<16} vs ref={ref!r} -> F1={f:.2f}")

pred='429 not 200'    vs ref='429 not 200' -> F1=1.00
pred='200 not 429'    vs ref='429 not 200' -> F1=1.00

Both score a perfect 1.00, yet one is right and one inverts the fact. Any metric built on unordered tokens is blind to this, and for a docs assistant that constantly deals in codes, numbers, and negations (“Pro is 600 rpm, not 60”), that blindness is dangerous.

It ignores meaning. F1 only credits tokens that match after normalization, so genuine synonyms earn nothing. "cannot" and "can't" normalize to different single tokens (stripping the apostrophe leaves cant, not cannot), so an answer that says "you cannot change the region" gets no overlap credit against a reference that says "you can't change the region" on those words – even though they mean the identical thing. The same happens with "25 dollars" versus "\$25", or "fixed at creation" versus "set once and immutable". F1 rewards lexical overlap, not semantic equivalence.

Practice Exercises#

Exercise 1: Predict then verify precision and recall#

Before running anything, predict the precision, recall, and F1 for Docent answering "The Pro plan costs 25 dollars per month" against the reference "25 dollars per month". Then call f1_score to check. Was precision or recall the one dragged down, and does that match the verbose-answer pattern from the lesson?

Exercise 2: Aggregate F1 over the golden set#

Score more than four answers at once. Take five of the canonical golden Q&A pairs from Module 2 (question, reference) and a matching Docent prediction for each – some exact, some verbose, some partial – then compute the mean F1 and the mean EM across all five. Confirm that mean F1 is at least as high as mean EM, and identify the single answer contributing the most to the gap between them.

Exercise 3: Expose the order-insensitivity failure yourself#

Construct your own Meridian example where token F1 gives a perfect 1.00 to a factually wrong answer purely because the tokens match. Use a reference that hinges on order or negation (rate limits, error codes, and plan comparisons are good sources), then write a wrong prediction that reuses the exact same words in a different arrangement. Explain in a sentence why no bag-of-words metric can catch it.

Summary#

Token-level F1 is the partial-credit metric that exact match cannot be. You represent both the prediction and the reference as bags of normalized tokens (reusing Lesson 1’s normalization), count the shared tokens, and compute precision (shared / prediction size, which punishes padding) and recall (shared / reference size, which punishes omission). Their harmonic mean is F1, and the harmonic mean matters because it stays low unless both precision and recall are high – so only a concise, complete answer scores well. Running a real f1_score on four Docent answers showed the payoff directly: an exact hit scores 1.0 under both metrics, but a verbose-but-correct answer (F1 = 0.25), a terse partial answer (F1 = 0.80), and a wrong answer (F1 = 0.0) are all a flat zero under exact match. Aggregated, the same four answers score mean EM = 0.25 but mean F1 = 0.51 – F1 recognizes how close Docent got instead of demanding perfection. The cost of that partial credit is a bag-of-words assumption that ignores word order ("200 not 429" scores identically to "429 not 200") and meaning (can't earns no credit against cannot), which is the ceiling that motivates the semantic and judge metrics later in the course.

Key Concepts#

• Bag of tokens – an answer represented as the multiset of its normalized words, discarding order; the object token F1 compares.
• Precision – $|\text{pred} \cap \text{ref}| / |\text{pred}|$; the fraction of predicted tokens that are correct. Low precision means padding or off-topic words.
• Recall – $|\text{pred} \cap \text{ref}| / |\text{ref}|$; the fraction of reference tokens the prediction covered. Low recall means missing facts.
• Token-level (SQuAD) F1 – the harmonic mean $2PR/(P+R)$; high only when precision and recall are both high, giving mostly-right answers a score between 0 and 1.
• Partial credit – F1’s key advantage over exact match: a verbose or terse but substantially-correct answer earns proportional credit instead of a flat zero (mean F1 0.51 vs mean EM 0.25 on this set).
• Bag-of-words blind spots – F1 ignores word order (opposite-meaning answers can tie at 1.0) and synonyms (can't ≠ cannot), so it measures lexical overlap, not correctness or meaning.

Why This Matters#

For question answering – Docent’s entire job – most answers are neither perfect strings nor total nonsense, so a metric that only recognizes perfection (exact match) throws away most of the signal you have. Token-level F1 is the standard the SQuAD benchmark and countless QA leaderboards use precisely because it grades how close an answer is, letting you rank two versions of Docent that both fail exact match but differ sharply in quality. Just as important is knowing where F1 lies: its blindness to order and meaning means a high F1 is necessary but not sufficient for a correct answer, and treating it as ground truth would let inverted-fact and synonym-heavy answers slip through. Deterministic overlap metrics like this one are the cheap, reproducible first pass of any eval suite – and understanding their ceiling is what tells you when to reach for the semantic and judge metrics that come later.

Continue Building Your Skills#

You upgraded from a switch to a dial: token-level F1 gives Docent’s near-correct answers the partial credit exact match refused them, lifting the same four-answer set from a mean EM of 0.25 to a mean F1 of 0.51 by rewarding shared tokens through precision and recall. You also saw the price of that partial credit – a bag of words that can’t see order or meaning, so "200 not 429" ties a perfect score with its own opposite. The next lesson keeps you in deterministic-metric territory but widens the lens from single tokens to n-grams: ROUGE and BLEU count overlapping word sequences, which recovers a little of the order information F1 discards and is the standard way to score longer, more free-form Docent answers. You’ll build both, run them for real, and find exactly where they still fall short for question answering.