Lesson 2 - Retrieval Metrics: Precision, Recall, MRR, NDCG

Welcome to Retrieval Metrics: Precision, Recall, MRR, NDCG#

In Lesson 1 you split Docent – our assistant for the Meridian serverless database docs – into the two systems it really is: a retriever that fetches doc pages and a generator that writes an answer from them. This lesson stays entirely on the retriever, because it sets the ceiling for everything downstream. If the retriever hands the model the wrong pages, no amount of prompt engineering saves the answer – the model is reasoning over the wrong text. So before we score a single generated word, we score the search: given a question, did the right Meridian page come back, and did it come back near the top?

The good news is that this measurement is completely deterministic. Docent’s retriever is pure Python – keyword overlap, no model, no API call – and our golden set maps each question to the one doc id that actually answers it. Compare the ranked ids the retriever returns against the known-relevant id and you get exact, reproducible numbers every run. This lesson builds the four metrics the information-retrieval field has used for decades – Recall@k, Precision@k, MRR, and NDCG@k – implements each for real, and turns them loose on Docent. Along the way one question from the golden set, “What does Meridian charge for the Free plan?”, reproduces the keyword-retriever bug you met in Module 1: its correct page never makes the cut, and the metrics put a number on exactly how much that costs.

In this lesson, you will:

• Define a golden set where each question has a known relevant Meridian doc id, and turn Docent’s retriever into a ranked list you can score
• Implement Recall@k / Hit@k, Precision@k, Mean Reciprocal Rank (MRR), and NDCG@k as real, deterministic functions
• Run all four over 9 golden questions and read the aggregates – Recall@2 = 0.889, Precision@2 = 0.444, MRR = 0.800, NDCG@2 = 0.807 – seeing how precision, recall, and rank-quality tell three different stories
• Watch the “Free plan” question drag Recall@2 from a perfect 1.000 down to 0.889, quantifying the Module-1 retrieval bug, and learn which metric to trust when

The setup: a golden set and a ranked list#

Every retrieval metric compares two things: the ranked list of documents the retriever returned, and the set of documents that are actually relevant to the question. Our golden set supplies the second half. Each entry pairs a question with the single Meridian doc id that contains its answer – "How much does the Pro plan cost per month?" is answered by the plans page, "What does a 401 error mean?" by the errors page, and so on. When there is exactly one relevant doc per question (the common case for a factual docs assistant), the math simplifies nicely, and we’ll lean on that.

For the returned half, we take Docent’s keyword retriever and ask it to rank all eight docs instead of just returning the top k. Ranking the full list lets us find the exact position of the relevant doc – rank 1, rank 5, wherever it lands – which every metric needs. The retriever scores each doc by how many words it shares with the question and sorts by that overlap, highest first.

import math

DOCS = [
    {"id": "auth", "title": "Authentication",
     "text": "Meridian authenticates every request with an API key sent in the Authorization header as a bearer token. "
             "Create keys in the dashboard under Settings > API Keys. Keys are either test or live mode; test keys only "
             "touch sandbox data. A key is shown once at creation and cannot be recovered, only rotated."},
    {"id": "ratelimits", "title": "Rate Limits",
     "text": "The Free plan allows 60 requests per minute. The Pro plan allows 600 requests per minute. Exceeding the "
             "limit returns HTTP 429 with a Retry-After header giving the seconds to wait. Rate limits are applied per "
             "API key, not per account."},
    {"id": "plans", "title": "Plans and Pricing",
     "text": "Meridian has three plans. Free costs 0 dollars per month and includes 1 GB of storage. Pro costs 25 dollars "
             "per month and includes 50 GB of storage. Scale costs 199 dollars per month and includes 500 GB of storage. "
             "All plans include unlimited read replicas."},
    {"id": "regions", "title": "Regions",
     "text": "Databases can be created in three regions: us-east, eu-west, and ap-south. The region is fixed at creation "
             "time and cannot be changed later; to move data to another region you must export and re-import into a new "
             "database. Cross-region read replicas are available on the Scale plan only."},
    {"id": "errors", "title": "Error Codes",
     "text": "Meridian uses standard HTTP status codes. 400 means a malformed request, 401 means a missing or invalid API "
             "key, 403 means the key is valid but lacks permission, 404 means the resource does not exist, and 429 means "
             "the rate limit was exceeded. 5xx codes indicate a server-side error and should be retried with backoff."},
    {"id": "backups", "title": "Backups",
     "text": "Automatic daily backups are retained for 7 days on the Free plan, 30 days on Pro, and 90 days on Scale. "
             "Backups run at 03:00 UTC. Point-in-time recovery to any second in the retention window is available on the "
             "Scale plan. Restoring a backup creates a new database and never overwrites the original."},
    {"id": "sdks", "title": "SDKs",
     "text": "Official SDKs are available for Python, JavaScript, and Go. Install the Python SDK with 'pip install meridian'. "
             "The SDK reads the MERIDIAN_API_KEY environment variable by default. All SDKs retry idempotent requests up to "
             "3 times on 5xx errors with exponential backoff."},
    {"id": "deletion", "title": "Deleting Data",
     "text": "Deleting a database is immediate and irreversible once the 24-hour grace period ends. During the grace period "
             "the database is soft-deleted and can be restored from the dashboard. After 24 hours the data and its backups "
             "are permanently purged. Deletion requires a live-mode key with admin permission."},
]

def retrieve_ranked(question, docs):
    """Return ALL doc ids ranked by keyword overlap, best first (Docent's retriever)."""
    q = set(question.lower().split())
    ranked = sorted(docs,
                    key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())),
                    reverse=True)
    return [d["id"] for d in ranked]

# Golden set: each question maps to the ONE doc id that actually answers it.
GOLDEN = [
    ("What HTTP status code does Meridian return when you exceed the rate limit?", "ratelimits"),
    ("How many requests per minute does the Pro plan allow?", "ratelimits"),
    ("How much does the Pro plan cost per month?", "plans"),
    ("Can you change a database's region after it is created?", "regions"),
    ("How long are backups retained on the Scale plan?", "backups"),
    ("Which environment variable does the Python SDK read for the API key?", "sdks"),
    ("What does a 401 error mean?", "errors"),
    ("How long is the grace period before a deleted database is permanently purged?", "deletion"),
    ("What does Meridian charge for the Free plan?", "plans"),   # the Module-1 bug
]

print(f"{'relevant':<11}{'rank':>5}   top-3 returned")
print("-" * 52)
for question, relevant in GOLDEN:
    ranked = retrieve_ranked(question, DOCS)
    rank = ranked.index(relevant) + 1
    print(f"{relevant:<11}{rank:>5}   {ranked[:3]}")

relevant    rank   top-3 returned
----------------------------------------------------
ratelimits     2   ['errors', 'ratelimits', 'auth']
ratelimits     1   ['ratelimits', 'plans', 'regions']
plans          2   ['ratelimits', 'plans', 'regions']
regions        1   ['regions', 'deletion', 'auth']
backups        1   ['backups', 'regions', 'sdks']
sdks           1   ['sdks', 'errors', 'auth']
errors         1   ['errors', 'auth', 'ratelimits']
deletion       1   ['deletion', 'backups', 'auth']
plans          5   ['errors', 'backups', 'auth']

The rank column is the whole story in miniature. Six of the nine questions put the relevant doc at rank 1 – the retriever nailed them. Two land at rank 2: the rate-limit-status question ranks errors above ratelimits (both talk about 429), and the Pro-cost question ranks ratelimits above plans (both contain “Pro plan”). Those are recoverable if we retrieve two docs. But the last row is the bug: the “Free plan” question buries its relevant plans page all the way down at rank 5, behind errors, backups, auth, and ratelimits. The word “free” appears in both ratelimits and backups, and generic words like “what” and “does” match everywhere, so keyword overlap simply ranks the right page too low to ever be retrieved. Hold onto that rank-5 – every metric below reacts to it differently.

Recall@k and Precision@k: did we get it, and how much noise came with it#

The first two metrics both look only at the top k documents – the ones Docent would actually feed the model – but they ask opposite questions. Let $R$ be the set of relevant docs and $\text{retrieved}_k$ the top-k ids.

Recall@k (also called Hit@k when there’s a single relevant doc) asks: did a relevant document make it into the top k at all? It’s the fraction of the relevant set you managed to retrieve:

Precision@k asks the mirror question: of the k documents we retrieved, what fraction were relevant? It measures how much of the context budget was well spent versus wasted on noise:

For a factual docs assistant with one relevant doc per question, these two behave very differently, and it’s worth seeing why up front. Recall@k is either 0 or 1 – either the single relevant doc is in the top k or it isn’t. Precision@k, on the other hand, can never exceed $1/k$: if only one of the k retrieved docs can be relevant, then at k = 2 the best possible precision is 0.5, at k = 3 it’s 0.333, and so on. That ceiling isn’t Docent failing; it’s an artifact of the task having a single right answer. Keep it in mind when the precision number looks alarmingly low.

def recall_at_k(ranked, relevant, k):
    return 1.0 if relevant in ranked[:k] else 0.0

def precision_at_k(ranked, relevant, k):
    return sum(1 for d in ranked[:k] if d == relevant) / k

MRR and NDCG: how high was the right doc ranked#

Recall@k and Precision@k are blunt about position: a relevant doc at rank 1 and a relevant doc at rank k score identically. But rank matters – a model reads the first passage more carefully than the last, and a good retriever puts the answer on top. Two metrics reward that.

Mean Reciprocal Rank (MRR) looks at the rank of the first relevant doc for each question and averages its reciprocal across all $N$ questions:

Reciprocal rank is 1.0 when the answer is at rank 1, 0.5 at rank 2, 0.2 at rank 5. Crucially, MRR is not capped at k – it sees the relevant doc wherever it lands in the full ranking, so it still credits the “Free plan” question a small 1/5 = 0.20 even though Recall@2 zeroed it. MRR answers “on average, how high is the first right doc?”

NDCG@k (Normalized Discounted Cumulative Gain) is the most complete of the four. It sums a gain for each relevant doc in the top k, discounted by how far down it sits, then normalizes by the best ordering possible. Discounted Cumulative Gain is:

where $rel_i$ is the relevance of the doc at position $i$ (here 1 if it’s the relevant doc, 0 otherwise). The $\log_2(i+1)$ denominator is the discount: rank 1 divides by $\log_2 2 = 1$, rank 2 by $\log_2 3 \approx 1.585$, rank 3 by $\log_2 4 = 2$. To make scores comparable across questions you divide by the IDCG – the DCG of the ideal ranking, which for a single relevant doc is just that doc at position 1, giving IDCG = 1. So:

With one relevant doc, NDCG@k is 1.0 if it’s at rank 1, about 0.63 at rank 2, 0.5 at rank 3, and 0 once it falls past k. Here are all four metrics as real functions:

def reciprocal_rank(ranked, relevant):
    return 1.0 / (ranked.index(relevant) + 1)

def ndcg_at_k(ranked, relevant, k):
    dcg = sum((1.0 if d == relevant else 0.0) / math.log2(i + 1)
              for i, d in enumerate(ranked[:k], start=1))
    idcg = 1.0 / math.log2(1 + 1)          # one relevant doc, ideal rank = 1
    return dcg / idcg

K = 2
print(f"per-question metrics at k={K}")
print(f"{'relevant':<11}{'rank':>5}{'R@2':>7}{'P@2':>7}{'RR':>7}{'NDCG@2':>8}")
print("-" * 45)
for question, relevant in GOLDEN:
    ranked = retrieve_ranked(question, DOCS)
    r = recall_at_k(ranked, relevant, K)
    p = precision_at_k(ranked, relevant, K)
    rr = reciprocal_rank(ranked, relevant)
    n = ndcg_at_k(ranked, relevant, K)
    print(f"{relevant:<11}{ranked.index(relevant)+1:>5}{r:>7.2f}{p:>7.2f}{rr:>7.2f}{n:>8.3f}")

per-question metrics at k=2
relevant    rank    R@2    P@2     RR  NDCG@2
---------------------------------------------
ratelimits     2   1.00   0.50   0.50   0.631
ratelimits     1   1.00   0.50   1.00   1.000
plans          2   1.00   0.50   0.50   0.631
regions        1   1.00   0.50   1.00   1.000
backups        1   1.00   0.50   1.00   1.000
sdks           1   1.00   0.50   1.00   1.000
errors         1   1.00   0.50   1.00   1.000
deletion       1   1.00   0.50   1.00   1.000
plans          5   0.00   0.00   0.20   0.000

Read the two rank-2 rows against the rank-1 rows and you can see the metrics separate. A rank-1 hit scores R@2 = 1, RR = 1.00, NDCG@2 = 1.000 – perfect on every rank-aware metric. A rank-2 hit still gets full Recall@2 (it’s in the top 2), but its RR drops to 0.50 and its NDCG@2 to 0.631, because those two metrics notice it wasn’t on top. And the “Free plan” row at the bottom is zero on everything measured at k=2, yet its reciprocal rank is still 0.20 – the only metric that refuses to pretend the doc doesn’t exist.

The aggregates: four numbers, four stories#

Now average each metric across the whole golden set, and sweep k from 1 to 3 so you can see the trade-offs move. MRR is reported once because it doesn’t depend on k – it reads the full ranking.

def mean(xs):
    return sum(xs) / len(xs)

for K in (1, 2, 3):
    ranks = [retrieve_ranked(q, DOCS) for q, _ in GOLDEN]
    rels = [rel for _, rel in GOLDEN]
    recall = mean([recall_at_k(rk, rel, K) for rk, rel in zip(ranks, rels)])
    prec = mean([precision_at_k(rk, rel, K) for rk, rel in zip(ranks, rels)])
    ndcg = mean([ndcg_at_k(rk, rel, K) for rk, rel in zip(ranks, rels)])
    print(f"k={K}   Recall@{K}={recall:.3f}   Precision@{K}={prec:.3f}   NDCG@{K}={ndcg:.3f}")

mrr = mean([reciprocal_rank(retrieve_ranked(q, DOCS), rel) for q, rel in GOLDEN])
print(f"MRR (rank-independent) = {mrr:.3f}")

# What does the Free-plan bug cost us? Recompute Recall@2 with it dropped.
good = [(q, rel) for q, rel in GOLDEN if q != "What does Meridian charge for the Free plan?"]
r_all = mean([recall_at_k(retrieve_ranked(q, DOCS), rel, 2) for q, rel in GOLDEN])
r_drop = mean([recall_at_k(retrieve_ranked(q, DOCS), rel, 2) for q, rel in good])
print(f"Recall@2 with Free-plan question   = {r_all:.3f}  ({len(GOLDEN)} questions)")
print(f"Recall@2 without Free-plan question = {r_drop:.3f}  ({len(good)} questions)")

k=1   Recall@1=0.667   Precision@1=0.667   NDCG@1=0.667
k=2   Recall@2=0.889   Precision@2=0.444   NDCG@2=0.807
k=3   Recall@3=0.889   Precision@3=0.296   NDCG@3=0.807
MRR (rank-independent) = 0.800
Recall@2 with Free-plan question   = 0.889  (9 questions)
Recall@2 without Free-plan question = 1.000  (8 questions)

Four metrics, and they genuinely disagree about how good this retriever is:

• Recall@2 = 0.889 says: “we usually get the right doc.” Eight of nine questions have their relevant page in the top 2. That’s the number most people care about first – did we even give the model a chance?
• Precision@2 = 0.444 looks bad, but it’s supposed to. With one relevant doc per question, precision is capped at 1/k = 0.5, and 0.444 is close to that ceiling – it means almost every top-2 slot that could be relevant was. Precision falls further at k=3 (0.296) purely because we retrieved a third doc that, by definition, can’t be relevant. Reading Precision@k without knowing the ceiling would badly mislead you.
• MRR = 0.800 says: “when the right doc is there, it’s ranked high.” The two rank-2 hits and one rank-5 miss pull it below 1.0, but 0.800 is a strong ranking signal – most answers are at or near the top.
• NDCG@2 = 0.807 agrees with MRR that ranking is mostly good, and unlike MRR it also penalizes the rank-5 miss to zero within the top-2 window, so it blends “did we get it” with “how high.”

And notice the k = 1 row: Recall@1 = 0.667 because the two rank-2 hits vanish when you only take the top doc. That’s the retrieval knob in action – raising k from 1 to 2 recovered two questions (recall 0.667 -> 0.889) at the cost of precision (0.667 -> 0.444). Going to k = 3 bought nothing extra (recall flat at 0.889, the Free-plan doc is still at rank 5) while precision kept dropping. For this retriever, k = 2 is the sweet spot.

The last two lines isolate the Module-1 bug precisely. Drop the single “Free plan” question and Recall@2 jumps from 0.889 to a perfect 1.000 – that one broken retrieval is the entire recall gap. This is the payoff of measuring retrieval separately: the metric doesn’t just say “Docent is imperfect,” it points at the exact question, the exact doc (plans), and the exact cost (one-ninth of recall).

Determinism: run it twice, get the same numbers#

None of this touches a model or an API. The retriever is keyword overlap, the golden labels are fixed, and every metric is arithmetic – so unlike the live Claude scores elsewhere in this course, these numbers are byte-for-byte reproducible. Run the entire aggregate block a second time and confirm:

run2 = []
for K in (1, 2, 3):
    ranks = [retrieve_ranked(q, DOCS) for q, _ in GOLDEN]
    rels = [rel for _, rel in GOLDEN]
    run2.append((K,
                 round(mean([recall_at_k(rk, rel, K) for rk, rel in zip(ranks, rels)]), 3),
                 round(mean([precision_at_k(rk, rel, K) for rk, rel in zip(ranks, rels)]), 3),
                 round(mean([ndcg_at_k(rk, rel, K) for rk, rel in zip(ranks, rels)]), 3)))
print("run 2 (k, Recall@k, Precision@k, NDCG@k):")
for row in run2:
    print(" ", row)
print("  MRR =", round(mean([reciprocal_rank(retrieve_ranked(q, DOCS), rel) for q, rel in GOLDEN]), 3))

run 2 (k, Recall@k, Precision@k, NDCG@k):
  (1, 0.667, 0.667, 0.667)
  (2, 0.889, 0.444, 0.807)
  (3, 0.889, 0.296, 0.807)
  MRR = 0.8

Identical to the first run, exactly as a deterministic metric should be. That reproducibility is a feature: retrieval metrics give you a stable, model-free scorecard you can track across retriever changes – swap keyword overlap for embeddings and re-run, and any movement in these four numbers is real signal, not run-to-run noise.

Practice Exercises#

Exercise 1: Add Recall@3 and read the sweep#

Extend the per-question table to also print Recall@3 alongside Recall@2, then find the question whose relevance status changes between k=2 and k=3. Explain from the rank column why raising k to 3 did (or did not) rescue the “Free plan” question, and connect your answer to why the aggregate Recall@3 stayed flat at 0.889.

Exercise 2: Watch MRR and NDCG@k diverge on a rank-2 hit#

Take just the rate-limit-status question ("What HTTP status code does Meridian return when you exceed the rate limit?", relevant ratelimits, which the retriever ranks at position 2). Compute its reciprocal rank and its NDCG@2 by hand, then verify with the functions. Why is RR = 0.50 but NDCG@2 = 0.631 for the same rank-2 doc – what makes NDCG’s discount gentler than MRR’s?

Exercise 3: Make Precision@k break past its ceiling with two relevant docs#

The 0.5 precision ceiling only exists because each question has one relevant doc. Invent a golden entry with two relevant doc ids – for example a question about backup retention that both backups and plans help answer – generalize recall_at_k and precision_at_k to take a set of relevant ids, and show a case where Precision@2 reaches 1.0. Explain how having two relevant docs changes what a “good” precision number even means.

Summary#

Retrieval metrics score the search half of RAG before you ever judge an answer, and they’re fully deterministic because Docent’s retriever is pure Python and the golden set fixes the relevant doc id for each question. You built four of them for real. Recall@k / Hit@k asks whether a relevant doc made the top k at all – for Docent, Recall@2 = 0.889, meaning eight of nine questions got their page into the context. Precision@k asks what fraction of the top k was relevant – Precision@2 = 0.444, which looks weak until you realize precision is capped at 1/k = 0.5 when there’s one relevant doc per question, so it’s actually near-ceiling. MRR averages the reciprocal rank of the first relevant doc and reported 0.800, and NDCG@k blends “did we get it” with “how high” via a 1/log2(rank+1) discount, reporting NDCG@2 = 0.807 – both say the ranking is mostly strong. The “Free plan” question exposed the Module-1 keyword bug: its relevant plans page sits at rank 5, missed entirely at k=2, and dropping that one question lifts Recall@2 from 0.889 to a perfect 1.000 – the metrics locate the failure to a single question and a single doc. The k sweep showed the classic trade-off (recall rises and precision falls as k grows, with k=2 the sweet spot here), and re-running produced byte-identical numbers, the reproducibility that makes retrieval metrics a trustworthy scorecard across retriever changes.

Key Concepts#

• Golden set – questions each labeled with the known relevant doc id(s); the ground truth every retrieval metric compares the returned ranking against.
• Recall@k / Hit@k – $|R \cap \text{retrieved}_k| / |R|$; did a relevant doc reach the top k? With one relevant doc it’s simply 0 or 1. Docent: 0.889 at k=2.
• Precision@k – $|R \cap \text{retrieved}_k| / k$; what fraction of the top k was relevant. Capped at $1/k$ with a single relevant doc, so read it against that ceiling. Docent: 0.444 at k=2.
• MRR – $\frac{1}{N}\sum_i 1/\text{rank}_i$; rank of the first relevant doc, averaged. Not capped at k, so it still credits a rank-5 miss 0.20. Docent: 0.800.
• NDCG@k – $\text{DCG@k}/\text{IDCG@k}$ with $\text{DCG@k} = \sum_i rel_i / \log_2(i+1)$; rewards relevant docs and their position, normalized to the ideal ordering. Docent: 0.807 at k=2.
• k selection – raising k trades precision for recall; here k=1→2 recovered two rank-2 hits, k=2→3 gained nothing because the missed doc is at rank 5.

Why This Matters#

When Docent answers wrong, retrieval metrics tell you which half to fix. A low Recall@k means the right page never reached the model – a search problem no prompt change can solve (exactly the “Free plan” case, where plans was at rank 5 and unreachable at k=2). Good recall with bad answers, on the other hand, points at the generator, which the next lessons on faithfulness and groundedness tackle. Just as important is reading each metric in context: Precision@k of 0.444 sounds like a failing grade but is near-optimal for a single-relevant-doc task, while MRR and NDCG reward the ranking quality that a bare Recall@k hides. Choosing the right metric – recall for “did we get the doc at all,” MRR and NDCG for “is it ranked high enough for the model to use” – and the right k is the difference between a scorecard that diagnoses your retriever and one that merely grades it.

Continue Building Your Skills#

You now have a model-free scorecard for the search half of Docent: four classic retrieval metrics, each implemented for real and read in context. Recall@2 (0.889) told you the right doc usually arrives, Precision@2 (0.444) was near its structural ceiling, and MRR (0.800) with NDCG@2 (0.807) confirmed the ranking is mostly strong – while the “Free plan” question’s rank-5 miss quantified the Module-1 keyword bug down to a single one-ninth dent in recall. The next lesson crosses the boundary from retrieval to generation: even when the retriever hands over the right context, the model can still invent claims the docs never made. You’ll measure faithfulness and groundedness – whether every sentence in Docent’s answer is actually supported by the retrieved passages – and start separating “we fetched the wrong page” from “we had the right page and still made something up.”