Lesson 5 - Guided Project: A Full RAG Evaluation

Welcome to the Guided Project#

This is the capstone for Evaluating RAG, and it is where the whole module pays off. Over the last four lessons you learned that Docent is really two systems — a retriever and a generator — and that each fails in its own way: retrieval metrics (recall, precision, MRR) score whether the right Meridian doc came back, while faithfulness and answer relevance score whether the generator did something sensible with whatever context it got. In this project you wire both halves into a single harness, run it live on Docent, and produce the one artifact a team actually acts on: a per-question table that says, for every wrong answer, whether to go fix the search or fix the prompt.

That diagnosis is the deliverable. A bare quality score (“Docent is 80% right”) tells you there’s a problem but not where it lives, and the two problems have opposite fixes — reindex your corpus versus rewrite your instructions. So we deliberately combine a deterministic retrieval layer (recall@k on fixed ids gives the same number every run) with a live generation layer (a claude-haiku-4-5 judge scoring faithfulness and relevance, which wobbles a point run to run), and then apply one simple rule: if the relevant document was never retrieved, it’s a retrieval failure; if it was retrieved and the answer is still wrong, it’s a generation failure. You’ll watch Docent’s Module-1 keyword-retrieval bug resurface on a Free-plan question and get correctly labeled as retrieval — a false refusal you fix in the search, not the prompt.

By the end of this project, you will be able to:

• Build one evaluation record per golden question that captures the question, the retrieved doc ids, the known relevant id, and Docent’s live answer
• Compute recall@k, precision@k, and MRR over the set deterministically, and confirm the numbers are identical on a re-run
• Score faithfulness and answer relevance for each answer with a live claude-haiku-4-5 judge, reusing the judge pattern from Lessons 3 and 4
• Combine retrieval and generation signals into a per-question diagnosis that classifies each failure as a retrieval or a generation problem — and read off what to fix

Stage 1: Set Up the Golden Set and Capture Evaluation Records#

A RAG evaluation needs one thing a plain generation eval doesn’t: for each question, the id of the document that actually contains the answer. That single label is what lets us later ask “did the retriever even fetch it?” — and it’s the difference between guessing and diagnosing. So our golden set is the usual (question, reference) pairs plus a rel field naming the known-relevant Meridian doc.

We reuse the canonical Docent verbatim — keyword-overlap retrieve() over the eight Meridian docs, then one claude-haiku-4-5 generation. For each question we run the pipeline and store an evaluation record: the question, the ids Docent retrieved, the relevant id we labeled, and the answer it produced. Every downstream stage reads from these records, so we build them once. The fifth question is planted on purpose — it’s a Free-plan rate-limit question phrased the way a real user might type it (“per-minute request cap”, “pricing tier”), and it’s the one that will expose the retriever.

import warnings; warnings.filterwarnings("ignore")
import json, re
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

DOCS = [
    {"id": "auth", "title": "Authentication",
     "text": "Meridian authenticates every request with an API key sent in the Authorization header as a bearer token. "
             "Create keys in the dashboard under Settings > API Keys. Keys are either test or live mode; test keys only "
             "touch sandbox data. A key is shown once at creation and cannot be recovered, only rotated."},
    {"id": "ratelimits", "title": "Rate Limits",
     "text": "The Free plan allows 60 requests per minute. The Pro plan allows 600 requests per minute. Exceeding the "
             "limit returns HTTP 429 with a Retry-After header giving the seconds to wait. Rate limits are applied per "
             "API key, not per account."},
    {"id": "plans", "title": "Plans and Pricing",
     "text": "Meridian has three plans. Free costs 0 dollars per month and includes 1 GB of storage. Pro costs 25 dollars "
             "per month and includes 50 GB of storage. Scale costs 199 dollars per month and includes 500 GB of storage. "
             "All plans include unlimited read replicas."},
    {"id": "regions", "title": "Regions",
     "text": "Databases can be created in three regions: us-east, eu-west, and ap-south. The region is fixed at creation "
             "time and cannot be changed later; to move data to another region you must export and re-import into a new "
             "database. Cross-region read replicas are available on the Scale plan only."},
    {"id": "errors", "title": "Error Codes",
     "text": "Meridian uses standard HTTP status codes. 400 means a malformed request, 401 means a missing or invalid API "
             "key, 403 means the key is valid but lacks permission, 404 means the resource does not exist, and 429 means "
             "the rate limit was exceeded. 5xx codes indicate a server-side error and should be retried with backoff."},
    {"id": "backups", "title": "Backups",
     "text": "Automatic daily backups are retained for 7 days on the Free plan, 30 days on Pro, and 90 days on Scale. "
             "Backups run at 03:00 UTC. Point-in-time recovery to any second in the retention window is available on the "
             "Scale plan. Restoring a backup creates a new database and never overwrites the original."},
    {"id": "sdks", "title": "SDKs",
     "text": "Official SDKs are available for Python, JavaScript, and Go. Install the Python SDK with 'pip install meridian'. "
             "The SDK reads the MERIDIAN_API_KEY environment variable by default. All SDKs retry idempotent requests up to "
             "3 times on 5xx errors with exponential backoff."},
    {"id": "deletion", "title": "Deleting Data",
     "text": "Deleting a database is immediate and irreversible once the 24-hour grace period ends. During the grace period "
             "the database is soft-deleted and can be restored from the dashboard. After 24 hours the data and its backups "
             "are permanently purged. Deletion requires a live-mode key with admin permission."},
]
BY_ID = {d["id"]: d for d in DOCS}

def retrieve(question, docs, k=2):
    q = set(question.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())), reverse=True)
    return scored[:k]

def docent_answer(question, docs, model="claude-haiku-4-5"):
    ctx = "\n\n".join(f"[{d['id']}] {d['title']}\n{d['text']}" for d in retrieve(question, docs))
    prompt = (f"Answer the question using ONLY the context. If the context does not contain the answer, say "
              f"\"I don't have that in the docs.\" Be concise.\n\nContext:\n{ctx}\n\nQuestion: {question}")
    msg = client.messages.create(model=model, max_tokens=200,
                                 messages=[{"role": "user", "content": prompt}])
    return msg.content[0].text

# golden set: reference answer PLUS the id of the doc that actually contains it (`rel`)
GOLDEN = [
    dict(q="How much does the Pro plan cost per month?",                            ref="25 dollars per month",    rel="plans"),
    dict(q="How long are backups retained on the Scale plan?",                      ref="90 days",                 rel="backups"),
    dict(q="Which environment variable does the Python SDK read for the API key?",  ref="MERIDIAN_API_KEY",        rel="sdks"),
    dict(q="How long is the grace period before a deleted database is permanently purged?", ref="24 hours",         rel="deletion"),
    dict(q="What is the per-minute request cap on the Free plan pricing tier?",     ref="60 requests per minute",  rel="ratelimits"),
]

# one evaluation record per question: question, retrieved ids, relevant id, live answer
records = []
for item in GOLDEN:
    retrieved = [d["id"] for d in retrieve(item["q"], DOCS)]
    answer = docent_answer(item["q"], DOCS)
    records.append(dict(q=item["q"], ref=item["ref"], rel=item["rel"], retrieved=retrieved, answer=answer))

for r in records:
    print(f"Q: {r['q']}")
    print(f"   relevant={r['rel']:11} retrieved={r['retrieved']}")
    print(f"   answer  : {r['answer']}\n")

Q: How much does the Pro plan cost per month?
   relevant=plans       retrieved=['ratelimits', 'plans']
   answer  : The Pro plan costs $25 per month.

Q: How long are backups retained on the Scale plan?
   relevant=backups     retrieved=['backups', 'regions']
   answer  : Backups are retained for 90 days on the Scale plan.

Q: Which environment variable does the Python SDK read for the API key?
   relevant=sdks        retrieved=['sdks', 'errors']
   answer  : The Python SDK reads the `MERIDIAN_API_KEY` environment variable by default.

Q: How long is the grace period before a deleted database is permanently purged?
   relevant=deletion    retrieved=['deletion', 'backups']
   answer  : The grace period is 24 hours. After 24 hours, the data and its backups are permanently purged.

Q: What is the per-minute request cap on the Free plan pricing tier?
   relevant=ratelimits  retrieved=['regions', 'backups']
   answer  : I don't have that in the docs.

Four of the five records look healthy — the relevant doc is in retrieved and the answer is right. The fifth is the tell. For the Free-plan question the relevant doc is ratelimits, but Docent retrieved ['regions', 'backups'] — ratelimits isn’t even in the list — and so, given two unrelated pages, Docent honestly said "I don't have that in the docs." That is a false refusal: the answer (60 requests per minute) is right there in the corpus; the retriever just never handed it over. Since this is a live run, Docent’s exact wording on the four good answers may shift slightly if you re-run, but the retrieved ids won’t — retrieval is deterministic — and the refusal on question five is stable.

Stage 2: Retrieval Metrics (Deterministic)#

Retrieval scoring needs no model — it’s set arithmetic over ids, so it’s fully reproducible. We compute three classics per question and average them. Recall@k asks whether the relevant doc landed in the top k (with one relevant doc it’s just 1 or 0). Precision@k is the fraction of the k retrieved docs that are relevant — a purity measure. MRR (mean reciprocal rank) rewards putting the relevant doc high: 1/rank if it’s found, 0 if it isn’t, so rank 1 scores 1.0 and rank 2 scores 0.5.

def recall_at_k(relevant_id, retrieved_ids):
    return 1.0 if relevant_id in retrieved_ids else 0.0

def precision_at_k(relevant_id, retrieved_ids):
    hits = sum(1 for r in retrieved_ids if r == relevant_id)
    return hits / len(retrieved_ids)

def reciprocal_rank(relevant_id, retrieved_ids):
    for i, r in enumerate(retrieved_ids, start=1):
        if r == relevant_id:
            return 1.0 / i
    return 0.0

def retrieval_report(records):
    rec  = [recall_at_k(r["rel"], r["retrieved"])     for r in records]
    prec = [precision_at_k(r["rel"], r["retrieved"])  for r in records]
    mrr  = [reciprocal_rank(r["rel"], r["retrieved"]) for r in records]
    return dict(recall=sum(rec)/len(rec), precision=sum(prec)/len(prec), mrr=sum(mrr)/len(mrr))

print(f"{'question':<46}{'recall':>7}{'MRR':>6}")
for r in records:
    print(f"{r['q'][:44]:<46}{recall_at_k(r['rel'], r['retrieved']):>7.2f}"
          f"{reciprocal_rank(r['rel'], r['retrieved']):>6.2f}")

agg = retrieval_report(records)
print("-" * 59)
print(f"aggregate  recall@2={agg['recall']:.2f}  precision@2={agg['precision']:.2f}  MRR={agg['mrr']:.2f}")

# deterministic: run it again, get the identical numbers
agg2 = retrieval_report(records)
print(f"re-run     recall@2={agg2['recall']:.2f}  precision@2={agg2['precision']:.2f}  MRR={agg2['mrr']:.2f}")

question                                       recall   MRR
How much does the Pro plan cost per month?       1.00  0.50
How long are backups retained on the Scale pl    1.00  1.00
Which environment variable does the Python SD    1.00  1.00
How long is the grace period before a deleted    1.00  1.00
What is the per-minute request cap on the Free   0.00  0.00
-----------------------------------------------------------
aggregate  recall@2=0.80  precision@2=0.40  MRR=0.70
re-run     recall@2=0.80  precision@2=0.40  MRR=0.70

The aggregate is recall@2 = 0.80, precision@2 = 0.40, MRR = 0.70, and the second run prints them byte-for-byte identical — no model, no randomness, just id arithmetic. Two rows deserve a look. The Free-plan question is the obvious one: recall = 0, because ratelimits never made the top 2. That single zero is what drags the aggregate recall down to 0.80, and it’s the fingerprint of the retrieval bug. The subtler row is the Pro-plan question: recall is 1 (good, the answer came back) but MRR is only 0.50, because plans was retrieved at rank 2, behind ratelimits — the keyword scorer put a near-miss first. That’s exactly the kind of ranking weakness recall alone would hide and MRR surfaces. The low precision@2 of 0.40 is expected here and not alarming: with k=2 and only ever one relevant doc, the best any single question can score is 0.50, so a fleet of good retrievals still averages low — precision@2 is telling you the second slot is usually filler, which for a two-doc context window is fine.

Stage 3: Generation Metrics (Live Judge)#

Retrieval metrics graded the lookup; now we grade the answer, and for that we need a model. Reusing the judge pattern from Lessons 3 and 4 — reasoning-first, validated JSON, on claude-haiku-4-5 — we score two things. Faithfulness asks whether every claim in the answer is supported by the retrieved context (a hallucinated or contradicted claim scores low; a refusal makes no claims, so it’s trivially faithful). Answer relevance asks whether the answer actually addresses the question — deliberately not whether it’s factually complete, because faithfulness already owns correctness. Splitting them this way is what makes the diagnosis clean: a faithful-but-irrelevant answer and an unfaithful-but-relevant answer point at different bugs.

def parse_json(text):
    m = re.search(r"\{.*\}", text, re.DOTALL)     # first {...} block, tolerant of prose around it
    if not m:
        raise ValueError("no JSON object found in judge output")
    return json.loads(m.group(0))

FAITH_SYS = (
    "You grade FAITHFULNESS: whether every factual claim in an ANSWER is supported by the CONTEXT. "
    "Ignore the question; judge only whether the answer's claims are backed by the context. If the answer "
    "makes no factual claim (for example it says it doesn't have the information), treat it as trivially "
    "faithful and score 5. Reply with ONLY JSON {\"reasoning\": \"<one sentence>\", \"faithfulness\": 1-5} "
    "where 5 = every claim supported, 1 = the answer contradicts or invents claims not in the context."
)
REL_SYS = (
    "You grade ANSWER RELEVANCE: whether the ANSWER is on-topic and actually attempts to answer "
    "the QUESTION that was asked. Judge ONLY responsiveness, not factual accuracy or completeness "
    "(a separate faithfulness metric covers correctness). An on-topic answer that addresses the "
    "question scores 5 even if its facts might be wrong; a refusal, a non-answer, or an answer about "
    "a different topic scores low. Reply with ONLY JSON {\"reasoning\": \"<one sentence>\", "
    "\"relevance\": 1-5} where 5 = directly answers this question, 1 = does not answer it at all."
)

def _score(system, user, key, model="claude-haiku-4-5"):
    msg = client.messages.create(model=model, max_tokens=150, system=system,
                                 messages=[{"role": "user", "content": user}])
    data = parse_json(msg.content[0].text)
    return int(data[key]), data["reasoning"]

def faithfulness_score(answer, context):
    return _score(FAITH_SYS, f"CONTEXT:\n{context}\n\nANSWER:\n{answer}\n\nGrade faithfulness.", "faithfulness")

def relevance_score(question, answer):
    return _score(REL_SYS, f"QUESTION: {question}\n\nANSWER: {answer}\n\nGrade answer relevance.", "relevance")

def context_for(record):   # the exact text Docent was given for this question
    return "\n\n".join(f"[{i}] {BY_ID[i]['title']}: {BY_ID[i]['text']}" for i in record["retrieved"])

# score every record, and print two in full: a healthy answer and the false refusal
for r in records:
    r["faith"], r["faith_why"] = faithfulness_score(r["answer"], context_for(r))
    r["relv"],  r["relv_why"]  = relevance_score(r["q"], r["answer"])

for r in (records[0], records[4]):
    print(f"Q: {r['q']}")
    print(f"  answer      : {r['answer']}")
    print(f"  faithfulness: {r['faith']}/5 -- {r['faith_why']}")
    print(f"  relevance   : {r['relv']}/5 -- {r['relv_why']}\n")

Q: How much does the Pro plan cost per month?
  answer      : The Pro plan costs $25 per month.
  faithfulness: 5/5 -- The answer states the Pro plan costs $25 per month, which is directly supported by the [plans] section stating 'Pro costs 25 dollars per month'.
  relevance   : 5/5 -- The answer directly addresses the question asked and provides a specific price for the Pro plan per month.

Q: What is the per-minute request cap on the Free plan pricing tier?
  answer      : I don't have that in the docs.
  faithfulness: 5/5 -- The answer makes no factual claim and states it lacks information, which is trivially faithful regardless of whether information actually exists in the context.
  relevance   : 2/5 -- The answer directly addresses the question but provides a non-answer by stating the information is unavailable, rather than answering the specific question asked.

Look closely at the Free-plan row, because it is the whole point of separating these two metrics. The refusal scored faithfulness 5 — and rightly so, it invents nothing — but relevance 2, because it doesn’t actually answer the question. If we only measured faithfulness, this false refusal would look perfect; it’s the relevance score that flags it as a failure at all. The healthy Pro-plan answer, by contrast, scores 5 on both. This is a live judge, so the exact numbers can wobble between runs — across our own re-runs the refusal’s relevance landed anywhere from 1 to 3, and the rationale wording changes each time — but the shape is rock stable: healthy answers score 5 on both, the refusal always scores high on faithfulness and lands well below the pass line on relevance, and (as the next stage relies on) it never once climbed to a passing 4.

Stage 4: Diagnose — Retrieval Problem or Generation Problem?#

Now we combine everything into the artifact this project exists to produce. For each record we already have the retrieval signal (recall@k) and the generation signals (faithfulness, relevance). We call an answer a failure if either generation score falls below 4, then apply one rule to every failure: if the relevant doc was not retrieved (recall@k = 0), it’s a RETRIEVAL problem — fix the search. If it was retrieved (recall@k = 1) and the answer is still wrong, it’s a GENERATION problem — fix the prompt. Same symptom, opposite cure.

PASS = 4   # a generation score below this counts as a failure

def is_failure(r):
    return r["faith"] < PASS or r["relv"] < PASS

def diagnose(r):
    if not is_failure(r):
        return "pass"
    return "RETRIEVAL" if r["rel"] not in r["retrieved"] else "GENERATION"

print(f"{'question':<40}{'rec':>4}{'faith':>6}{'relv':>5}  {'verdict':<7}diagnosis")
for r in records:
    rc = recall_at_k(r["rel"], r["retrieved"])
    print(f"{r['q'][:38]:<40}{rc:>4.0f}{r['faith']:>6}{r['relv']:>5}  "
          f"{('FAIL' if is_failure(r) else 'ok'):<7}{diagnose(r)}")

print("\n--- deliverable: what to fix ---")
for r in records:
    if is_failure(r):
        d = diagnose(r)
        fix = "the retriever, not the prompt" if d == "RETRIEVAL" else "the generation prompt, not the search"
        print(f"FAILURE: {r['q']}")
        print(f"  recall@2={recall_at_k(r['rel'], r['retrieved']):.0f}  faith={r['faith']}  relv={r['relv']}  ->  {d} problem  ->  fix {fix}")

question                                 rec faith relv  verdict diagnosis
How much does the Pro plan cost per mo     1     5    5  ok     pass
How long are backups retained on the S     1     5    5  ok     pass
Which environment variable does the Py     1     5    5  ok     pass
How long is the grace period before a      1     5    5  ok     pass
What is the per-minute request cap on      0     5    2  FAIL   RETRIEVAL

--- deliverable: what to fix ---
FAILURE: What is the per-minute request cap on the Free plan pricing tier?
  recall@2=0  faith=5  relv=2  ->  RETRIEVAL problem  ->  fix the retriever, not the prompt

There it is — the payoff of the whole module in one line. Docent’s single failure, the Free-plan false refusal, is diagnosed as a RETRIEVAL problem: recall@2 = 0 means ratelimits never reached the generator, so no amount of prompt-tuning could have saved this answer — the fix is in the search (better retrieval: fuzzy matching, embeddings, or a higher k), not the instructions. Notice how the signals cooperate to say this: faithfulness stayed at 5 (the model didn’t lie, it just abstained), relevance dropped because it didn’t answer, and recall@2 = 0 pinned the blame on retrieval. A team reading this table knows exactly which backlog the ticket goes in.

But our set only has a retrieval failure — every retrieved answer was fine — so let’s prove the other branch fires too. We feed the diagnoser one more record: same pipeline, but imagine the generator had botched a question whose context was perfectly good. We take the Scale-plan backups question (whose retrieve() correctly returns backups) and pair it with a wrong answer — “30 days,” which is the Pro number sitting right there in the same page — then score and diagnose it live.

# a "what-if" regression: right context retrieved, wrong answer generated
iq = "How long are backups retained on the Scale plan?"
whatif = dict(q=iq, ref="90 days", rel="backups",
              retrieved=[d["id"] for d in retrieve(iq, DOCS)],
              answer="Backups are retained for 30 days on the Scale plan.")   # 30 is Pro's number, not Scale's
whatif["faith"], fw = faithfulness_score(whatif["answer"], context_for(whatif))
whatif["relv"],  rw = relevance_score(whatif["q"], whatif["answer"])

print(f"Q: {whatif['q']}")
print(f"  retrieved   : {whatif['retrieved']}  (recall@2 = {recall_at_k(whatif['rel'], whatif['retrieved']):.0f})")
print(f"  answer      : {whatif['answer']}")
print(f"  faithfulness: {whatif['faith']}/5 -- {fw}")
print(f"  relevance   : {whatif['relv']}/5 -- {rw}")
print(f"  diagnosis   : {diagnose(whatif)}")

Q: How long are backups retained on the Scale plan?
  retrieved   : ['backups', 'regions']  (recall@2 = 1)
  answer      : Backups are retained for 30 days on the Scale plan.
  faithfulness: 1/5 -- The context states that backups are retained for 90 days on Scale (not 30 days), and 30 days is specified for the Pro plan. The answer contradicts the context.
  relevance   : 5/5 -- The answer directly addresses the question by providing a specific retention period (30 days) for backups on the Scale plan.
  diagnosis   : GENERATION

The mirror image, and the second branch confirmed. Here recall@2 = 1 — the backups page was retrieved, the generator had the “90 days on Scale” fact in front of it — yet the answer said 30 days, so faithfulness cratered to 1. The relevant doc came back and the answer is still wrong, so the diagnosis is GENERATION: this is a prompt or model problem, and reindexing the corpus would fix nothing. Set the two failures side by side and the discipline is obvious — identical symptom (“wrong answer”), and the recall@k signal is the fork that sends one ticket to the retrieval team and the other to the prompt. That is the diagnosis this whole module was building toward.

Practice Exercises#

Exercise 1: Confirm the retrieval fix would actually work#

You diagnosed the Free-plan failure as retrieval, but a diagnosis is a hypothesis until you test the fix. Re-run just the retrieval half for that question with a larger k (say k=4) and check whether ratelimits now appears in the retrieved ids. If it does, recompute recall@k for the set and confirm the number the fix buys you — without touching Docent’s prompt at all.

Exercise 2: Add a groundedness gate before you trust an answer#

Right now an answer “passes” if faithfulness and relevance are both at least 4. Add a third, hard rule from earlier in the module: an answer that makes factual claims must cite content actually present in its retrieved context. Write a cheap deterministic pre-check — before the judge even runs — that flags any answer whose key facts don’t appear in context_for(record), and see whether it catches the same failures the judge does.

Exercise 3: Break out a per-category diagnosis summary#

One diagnosed failure is a data point; a pattern is what tells you where to invest. Extend the harness so that, after diagnosing every record, it prints a summary: how many failures were RETRIEVAL versus GENERATION, and the overall pass rate. Then imagine scaling to fifty questions — which single number would you put on the dashboard to decide whether this sprint fixes the retriever or the prompt?

Summary#

You ran a full RAG evaluation on Docent and, more importantly, diagnosed where its failures come from. You built one evaluation record per golden question — question, retrieved ids, the known-relevant id, and Docent’s live answer — making the relevant-doc label the field that turns “wrong answer” into a root cause. You computed retrieval metrics deterministically (recall@2 = 0.80, precision@2 = 0.40, MRR = 0.70, identical on re-run) and watched the Free-plan question post recall = 0. You scored faithfulness and answer relevance with a live claude-haiku-4-5 judge, keeping them separate on purpose so a false refusal reads as faithful-but-irrelevant. And you combined the signals into the diagnosis that is the module’s deliverable: the Free-plan false refusal is a RETRIEVAL problem (recall = 0 — fix the search), while an answer that mangles context it did retrieve is a GENERATION problem (recall = 1, faithfulness = 1 — fix the prompt).

Key Concepts#

• Evaluation record — the per-question bundle of question, retrieved ids, rel (relevant id), and answer; the relevant-id label is what makes RAG failures diagnosable rather than merely detectable.
• Deterministic retrieval metrics — recall@k (was the right doc fetched), precision@k (how pure the retrieved set is), and MRR (how high the right doc ranked); computed from ids alone, so they reproduce exactly on every run.
• Faithfulness vs answer relevance — two separate live-judge scores: faithfulness catches invented or contradicted claims, relevance catches off-topic answers and false refusals; kept apart because they point at different bugs.
• Retrieval-vs-generation diagnosis — the rule that classifies each failure by recall@k: recall = 0 means the relevant doc never arrived (fix retrieval), recall = 1 with a wrong answer means the context was there and squandered (fix generation).
• False refusal — a faithful-but-irrelevant “I don’t have that” caused by the retriever missing the page; invisible to faithfulness alone, caught by relevance, and pinned to retrieval by recall.

Why This Matters#

In production a wrong RAG answer arrives as a single symptom — an unhappy user — and the two possible causes have completely different, non-overlapping fixes: reindex the corpus and tune retrieval, or rewrite the prompt and constrain the generator. Guess wrong and you spend a sprint on the healthy half of the system while the bug ships untouched. The harness you built here removes the guess: it measures both halves, keeps the signals separable, and applies one recall-based rule that routes every failure to the right team. That is the difference between a dashboard that says “quality is 80%” and one that says “the 20% is retrieval — here’s the ticket.” Every real RAG system you evaluate will need exactly this split-and-diagnose discipline, because a blended score is only ever a symptom, and you can’t fix a symptom.

Continue Building Your Skills#

You just built the harness a RAG team actually runs before every release: it measures the retriever and the generator separately, scores them with the right tools for each — deterministic id arithmetic for retrieval, a live judge for generation — and then does the thing a bare quality score never can, which is tell you which half broke. You watched Docent’s Module-1 keyword bug resurface as a Free-plan false refusal and get correctly labeled RETRIEVAL, and you proved the generation branch fires too. Carry the core instinct forward: when a RAG answer is wrong, don’t reach for the prompt reflexively — ask whether the relevant document ever arrived, because the answer to that question decides everything you do next. In Module 6, Docent grows the ability to act, and you’ll learn to evaluate systems where the failure can hide in any step of a chain, not just the last one.