Lesson 3 - ROUGE, BLEU & N-gram Metrics

Welcome to ROUGE, BLEU & N-gram Metrics#

In the last two lessons you scored Docent – our assistant for the Meridian serverless database docs – with exact match and token-level F1. Both compare a candidate answer to a reference, but neither cares about word order or gives much credit to a long, mostly-right answer that uses a different sentence shape. That’s exactly the gap two famous metrics were built to fill: ROUGE, which came from automatic summarization, and BLEU, which came from machine translation. Both score generated text by counting how many short word sequences – n-grams – it shares with a reference.

These metrics are everywhere in NLP papers and leaderboards, so you need to know how they work. But their origin is also their trap: they were designed for longer generated text (summaries, translations) where surface overlap is a reasonable proxy for quality. Point them at Docent’s short factual answers and they can quietly reward a fluent, confidently-wrong answer just because it reuses the reference’s vocabulary. In this lesson you’ll compute real ROUGE and BLEU scores on Docent answers, then watch a wrong answer beat a correct one – the clearest possible warning about when not to trust an n-gram metric.

In this lesson, you will:

• Build the n-gram intuition – unigrams, bigrams, and clipped overlap – by hand, so the library scores aren’t a black box
• Compute real ROUGE-1, ROUGE-2, and ROUGE-L (precision, recall, F) on Docent answers with the rouge-score library
• Compute BLEU with nltk and understand its n-gram precision and brevity penalty, and why it fits translation but not short QA
• See a concrete case where high ROUGE and high BLEU do not mean correct, and know when to reach for exact match, F1, or a judge instead

The n-gram intuition#

An n-gram is just a sliding window of n consecutive tokens. The 1-grams (unigrams) of "the rate limit" are the, rate, limit; the 2-grams (bigrams) are the rate and rate limit. ROUGE and BLEU both reduce “how similar are these two texts?” to “how many n-grams do they share?” – unigrams capture which words appear, and bigrams (and higher) capture a little bit of word order, since rate limit and limit rate are different bigrams.

The building block for BLEU-style precision is clipped overlap: for each n-gram in the candidate, you count it as a match at most as many times as it appears in the reference, so a candidate can’t game the score by repeating one correct word ten times. Here it is by hand – no library – on a small pair, so you can see exactly what the libraries do under the hood:

from collections import Counter

def ngrams(text, n):
    toks = text.lower().split()
    return [tuple(toks[i:i + n]) for i in range(len(toks) - n + 1)]

def clipped_precision(reference, candidate, n):
    ref_counts = Counter(ngrams(reference, n))
    cand_counts = Counter(ngrams(candidate, n))
    if not cand_counts:
        return 0.0
    # clip each candidate n-gram count by how many times it appears in the reference
    overlap = sum(min(c, ref_counts[g]) for g, c in cand_counts.items())
    return overlap / sum(cand_counts.values())

reference = "the rate limit returns http 429"
candidate = "http 429 is the rate limit response"

print("reference :", reference)
print("candidate :", candidate)
print()
print("reference unigrams:", [g[0] for g in ngrams(reference, 1)])
print("candidate unigrams:", [g[0] for g in ngrams(candidate, 1)])
print("reference bigrams :", [" ".join(g) for g in ngrams(reference, 2)])
print("candidate bigrams :", [" ".join(g) for g in ngrams(candidate, 2)])
print()
print(f"clipped unigram precision : {clipped_precision(reference, candidate, 1):.3f}")
print(f"clipped bigram  precision : {clipped_precision(reference, candidate, 2):.3f}")

reference : the rate limit returns http 429
candidate : http 429 is the rate limit response

reference unigrams: ['the', 'rate', 'limit', 'returns', 'http', '429']
candidate unigrams: ['http', '429', 'is', 'the', 'rate', 'limit', 'response']
reference bigrams : ['the rate', 'rate limit', 'limit returns', 'returns http', 'http 429']
candidate bigrams : ['http 429', '429 is', 'is the', 'the rate', 'rate limit', 'limit response']

clipped unigram precision : 0.714
clipped bigram  precision : 0.500

Five of the candidate’s seven unigrams appear in the reference, so unigram precision is 5/7 = 0.714. Only three of its six bigrams match (http 429, the rate, rate limit), so bigram precision drops to 3/6 = 0.500. Notice how the bigram score punishes the reordering that the unigram score ignored: the words are almost all there, but the sequence is different. That difference between “which words” and “in what order” is the whole reason both a unigram and a bigram (or longer) term exist inside these metrics.

ROUGE: recall-oriented, built for summarization#

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) was designed to score summaries: given a human reference summary, how much of its content did the machine summary recover? Because the priority is coverage, ROUGE leans on recall – what fraction of the reference’s n-grams show up in the candidate – though the standard rouge-score library reports precision, recall, and F-measure for each variant. The three variants you’ll see most:

• ROUGE-1 – unigram overlap. Did the candidate use the right words?
• ROUGE-2 – bigram overlap. Did it use the right adjacent word pairs? This is more sensitive to fluency and order.
• ROUGE-L – based on the longest common subsequence (LCS), the longest run of tokens that appears in both texts in the same order but not necessarily adjacent. ROUGE-L rewards in-order overlap without demanding the words be contiguous, so it captures sentence-level structure more gracefully than a fixed bigram.

For recall and precision, ROUGE-N is defined as:

and the F-measure combines them the usual way. Let’s compute the real thing with the rouge-score library on a genuine Docent scenario. We take one of Meridian’s longer-form questions – “Can you change a database’s region after it is created?” – whose reference answer is a full sentence, and score three different candidate answers against it: a good answer, a correct paraphrase, and a fluent but wrong answer that reuses the reference’s words while flipping the meaning.

import warnings
warnings.filterwarnings("ignore")
from rouge_score import rouge_scorer

question = "Can you change a database's region after it is created?"
reference = ("No; the region is fixed at creation and you must export and "
             "re-import into a new database to move regions.")

candidates = {
    "A. good answer (correct, lots of overlap)":
        ("No, the region is fixed at creation time and cannot be changed; to move "
         "regions you must export the data and re-import it into a new database."),
    "B. paraphrase (correct, different words)":
        ("A database's location is permanent once it is set up, so relocating it means "
         "dumping the data and loading it into a fresh instance elsewhere."),
    "C. wrong answer (INCORRECT, but heavy word overlap)":
        ("Yes, the region is not fixed at creation and you do not need to export and "
         "re-import into a new database to move regions."),
}

scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)

def show(name, cand):
    s = scorer.score(reference, cand)
    print(name)
    for m in ("rouge1", "rouge2", "rougeL"):
        r = s[m]
        print(f"  {m:8s}  P={r.precision:.3f}  R={r.recall:.3f}  F={r.fmeasure:.3f}")
    print()

print("Reference:", reference)
print("=" * 68)
for name, cand in candidates.items():
    print("Candidate", name.split('.')[0] + ":", cand)
    show(name, cand)

Reference: No; the region is fixed at creation and you must export and re-import into a new database to move regions.
====================================================================
Candidate A: No, the region is fixed at creation time and cannot be changed; to move regions you must export the data and re-import it into a new database.
A. good answer (correct, lots of overlap)
  rouge1    P=0.750  R=1.000  F=0.857
  rouge2    P=0.556  R=0.750  F=0.638
  rougeL    P=0.643  R=0.857  F=0.735

Candidate B: A database's location is permanent once it is set up, so relocating it means dumping the data and loading it into a fresh instance elsewhere.
B. paraphrase (correct, different words)
  rouge1    P=0.231  R=0.286  F=0.255
  rouge2    P=0.040  R=0.050  F=0.044
  rougeL    P=0.154  R=0.190  F=0.170

Candidate C: Yes, the region is not fixed at creation and you do not need to export and re-import into a new database to move regions.
C. wrong answer (INCORRECT, but heavy word overlap)
  rouge1    P=0.760  R=0.905  F=0.826
  rouge2    P=0.667  R=0.800  F=0.727
  rougeL    P=0.760  R=0.905  F=0.826

Read those three blocks carefully, because they contain the whole point of the lesson. The good answer A scores high (ROUGE-L F = 0.735) – exactly what you want. The correct paraphrase B, which says the same true thing in completely different words, collapses to ROUGE-L F = 0.170, because it barely shares any n-grams with the reference even though it is right. And the wrong answer C – which says the opposite of the truth by inserting “not” and “do not” – scores ROUGE-L F = 0.826, higher than the correct answer A and nearly five times the correct paraphrase B. We’ll come back to why that is so dangerous; first, BLEU.

BLEU: precision-oriented, built for translation#

BLEU (Bilingual Evaluation Understudy) came from machine translation, where the question is “how close is this translation to a reference translation?” Where ROUGE leans on recall, BLEU is built on precision: of the n-grams the candidate produced, how many are in the reference? It combines clipped n-gram precisions for n = 1..4 (usually a geometric mean) and multiplies by a brevity penalty – a factor below 1 that punishes candidates shorter than the reference, so a translation can’t win by emitting just the two words it’s most sure of. Written out, roughly:

where $p_n$ is the clipped precision for n-grams of length $n$, the $w_n$ are weights (uniform by default), and BP is the brevity penalty. Because a short candidate often has zero 3-grams or 4-grams in common with the reference, a raw BLEU would collapse to zero; in practice a smoothing function nudges those zero counts up so the score stays informative. We use nltk’s sentence_bleu with smoothing:

import warnings
warnings.filterwarnings("ignore")
from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction

smooth = SmoothingFunction().method1  # avoids a hard zero when a higher-order n-gram is missing

def bleu(reference, candidate):
    ref_tokens = [reference.lower().split()]   # BLEU takes a LIST of reference token lists
    cand_tokens = candidate.lower().split()
    return sentence_bleu(ref_tokens, cand_tokens, smoothing_function=smooth)

# --- Same longer-form region answer as the ROUGE demo ---
reference = ("No; the region is fixed at creation and you must export and "
             "re-import into a new database to move regions.")
candidates = {
    "A. good answer (correct)":
        ("No, the region is fixed at creation time and cannot be changed; to move "
         "regions you must export the data and re-import it into a new database."),
    "B. paraphrase (correct, different words)":
        ("A database's location is permanent once it is set up, so relocating it means "
         "dumping the data and loading it into a fresh instance elsewhere."),
    "C. wrong answer (INCORRECT, heavy overlap)":
        ("Yes, the region is not fixed at creation and you do not need to export and "
         "re-import into a new database to move regions."),
}
print("BLEU on the longer region answer (ref =", len(reference.split()), "tokens)")
print("-" * 60)
for name, cand in candidates.items():
    print(f"  {name:44s} BLEU={bleu(reference, cand):.3f}")

# --- Short factoid QA: where n-gram metrics fall apart ---
print()
print("BLEU on a SHORT factoid answer (ref = 'twenty-five dollars')")
print("-" * 60)
short_ref = "25 dollars per month"
short_cases = {
    "correct, other words : 'twenty-five USD each month'": "twenty-five USD each month",
    "flat WRONG           : '250 dollars per month'": "250 dollars per month",
}
for label, cand in short_cases.items():
    print(f"  {label:52s} BLEU={bleu(short_ref, cand):.3f}")

BLEU on the longer region answer (ref = 20 tokens)
------------------------------------------------------------
  A. good answer (correct)                     BLEU=0.299
  B. paraphrase (correct, different words)     BLEU=0.020
  C. wrong answer (INCORRECT, heavy overlap)   BLEU=0.582

BLEU on a SHORT factoid answer (ref = 'twenty-five dollars')
------------------------------------------------------------
  correct, other words : 'twenty-five USD each month'  BLEU=0.080
  flat WRONG           : '250 dollars per month'       BLEU=0.398

BLEU tells the same uncomfortable story as ROUGE, only louder. On the region answer, the wrong answer C scores 0.582 – nearly double the correct answer A at 0.299, and 29 times the correct paraphrase B at 0.020. And the short-factoid block is the real gut-punch: a correct answer phrased as "twenty-five USD each month" scores 0.080, while a flatly wrong answer of "250 dollars per month" – off by a factor of ten – scores 0.398 because it happens to share the tokens dollars, per, and month with the reference. On short factual answers, BLEU is measuring vocabulary bingo, not correctness.

Why they mislead for QA – and what to use instead#

Both demos failed the same way, and it’s worth naming the mechanism precisely: n-gram metrics reward surface overlap, not meaning. They have no idea that inserting “not” flips a sentence from true to false, and no idea that "twenty-five USD" and "25 dollars" mean the same thing. So they systematically make two opposite errors on QA:

• False positives: a fluent wrong answer that recycles the reference’s words scores high (answer C, "250 dollars per month"). This is the dangerous one – your eval says “great” about an answer that would mislead a user.
• False negatives: a correct answer in fresh words scores low (paraphrase B, "twenty-five USD each month"). Your eval penalizes Docent for being right in its own words, which pushes you to optimize for parroting the reference rather than being correct.

Why do these metrics work fine for summarization and translation, then? Because there the text is long, and there’s often no single “right” wording, so aggregate n-gram overlap across many words really does correlate with quality – a summary that shares most of a reference’s content genuinely is a decent summary, and one wrong word can’t flip the meaning of a whole paragraph the way it flips a one-line answer. The correlation is statistical and it holds at length. Short factoid QA is the worst case: the answer is one fact, a single token can invert it, and there are many correct ways to phrase it.

So the practical rule for Docent-style QA:

• For short factual answers, prefer exact match with normalization (Lesson 1) or token-level F1 (Lesson 2) – and better still, a targeted check like “does the answer contain 429?” that tests the fact, not the wording.
• For longer generated text (a multi-sentence explanation, a summary of a doc page), ROUGE-L is a reasonable, cheap signal – but treat it as a smoke alarm, not a verdict, and pair it with something meaning-aware.
• When you truly need to judge correctness of free-form text, none of these surface metrics suffice; that’s the job of the LLM-as-judge approach you’ll build in a later module.

ROUGE and BLEU earn their place in your toolkit precisely because they’re fast, deterministic, and free of any model call. Just use them where their assumption – that word overlap tracks quality – actually holds, and never let a high ROUGE number talk you out of checking whether the answer is true.

Practice Exercises#

Exercise 1: Make ROUGE-2 and ROUGE-L disagree with ROUGE-L#

Take the region reference from the ROUGE demo and write a new candidate that keeps all the same words as the reference but shuffles their order (for example, move the second clause in front of the first). Score it with the same rouge_scorer. Explain why ROUGE-1 stays high while ROUGE-2 drops, using the difference between “which words” and “which adjacent pairs.”

Exercise 2: Confirm the brevity penalty#

BLEU punishes answers that are too short. Using the bleu helper, score the region reference against a truncated candidate that keeps only its first few words (e.g. "No, the region is fixed at creation"), then against the full good answer A. The truncated answer is a correct prefix, yet BLEU should still rank it below the full answer. Explain which part of the BLEU formula causes that.

Exercise 3: Show a fact-check beating n-gram overlap#

The lesson argued that for short factual answers a targeted fact check beats ROUGE/BLEU. Prove it: for the rate-limit question (ref = "429"), write a tiny contains_fact(answer, key) that returns True when the key token appears in a normalized answer, and run it on a correct answer ("It returns HTTP 429."), a correct paraphrase ("The status is four twenty-nine." – treat as a hard case), and a wrong answer ("It returns HTTP 200."). Compare how the fact check labels each versus what ROUGE-1 would give them.

Summary#

ROUGE and BLEU score generated text by counting overlapping n-grams against a reference. ROUGE is recall-oriented and comes from summarization: ROUGE-1 (unigrams), ROUGE-2 (bigrams), and ROUGE-L (longest common subsequence, which is order-aware without demanding adjacency). BLEU is precision-oriented and comes from translation: clipped n-gram precisions for n = 1..4 combined with a brevity penalty, usually with smoothing so short candidates don’t collapse to zero. You computed both for real – ROUGE with the rouge-score library, BLEU with nltk – on three Docent answers to the same Meridian question. The results are the lesson: a fluent wrong answer that recycled the reference’s words scored highest on both metrics (ROUGE-L F = 0.826, BLEU = 0.582), beating a correct answer (0.735, 0.299) and crushing a correct paraphrase (0.170, 0.020). On a short factoid, a flatly wrong "250 dollars per month" even out-scored a correct "twenty-five USD each month". These metrics measure surface overlap, not truth, so they fit longer generated text and are a poor, sometimes actively misleading, fit for short factual QA – where exact match, token-level F1, a targeted fact check, or a judge is the right tool.

Key Concepts#

• N-gram – a sliding window of n consecutive tokens; unigrams capture which words appear, bigrams and higher capture some word order.
• ROUGE – recall-oriented overlap for summarization; ROUGE-1/ROUGE-2 (n-gram overlap) and ROUGE-L (longest common subsequence, order-aware). The rouge-score library reports precision, recall, and F.
• BLEU – precision-oriented overlap for translation; clipped n-gram precision for n = 1..4 times a brevity penalty, with smoothing to avoid zeros on short text.
• Clipped precision – a candidate n-gram counts at most as many times as it appears in the reference, so repetition can’t inflate the score.
• Surface overlap is not correctness – both metrics reward shared words, so a wrong answer that reuses reference vocabulary scores high and a correct paraphrase scores low.
• Fit to length – n-gram overlap correlates with quality for long text (summaries, translations) but misleads for short factoid QA, where EM, F1, or a judge is better.

Why This Matters#

ROUGE and BLEU dominate NLP leaderboards and show up in nearly every eval library, so you will be handed a “the model scores 0.6 ROUGE” number and asked whether that’s good. Knowing what these metrics actually measure – and running the demo where a wrong answer beats a correct one – is what stops you from shipping a Docent that games surface overlap while giving users false information. The deeper lesson generalizes: a metric is only trustworthy where its core assumption holds, and ROUGE/BLEU assume word overlap tracks meaning, which breaks precisely on the short, single-fact answers a docs assistant lives on. Use them for the long-form text they were built for, keep exact match and F1 for the factoids, and reserve real correctness judgments for a model-based judge – the tool the rest of the course builds toward.

Continue Building Your Skills#

You can now compute ROUGE and BLEU for real and, more importantly, you know when to distrust them: they reward shared words, so a wrong answer that recycles the reference’s vocabulary can top a correct paraphrase, and short factual answers are their weakest case. That makes them a fast, deterministic smoke signal for longer generated text and a poor judge of a one-line factoid. Next, in Lesson 4, you’ll leave surface overlap behind for a different kind of deterministic check – validating that Docent’s structured outputs are well-formed: that an answer meant to be JSON with an answer and a doc_id field actually parses, contains the required keys, and has values of the right type. Where ROUGE asks “how similar is this text?”, schema checks ask “is this the right shape?” – a question code can answer with total confidence.