Lesson 1 - The RAG Eval Problem: Two Systems in One

Welcome to the RAG Eval Problem#

For four modules you’ve evaluated Docent — our Meridian documentation assistant — mostly by looking at the answer it produces and asking “is that right?” That question is necessary, but for a retrieval-augmented system it is not enough, because Docent is not one system. It is two systems wired in series: a retriever that pulls the Meridian doc pages most relevant to a question, and a generator (Claude) that writes an answer using only those pages. A wrong final answer can come from either half — and if all you look at is the answer, you cannot tell which half broke.

This lesson makes that split concrete. We’ll decompose Docent into its two stages, lay out the four ways the pipeline can fail, and then reproduce — live and deterministically — a bug you first met in Module 1: Docent refusing to answer “What does Meridian charge for the Free plan?” even though the price is right there in the docs. You’ll see that this failure is entirely a retrieval problem wearing a generation costume, and that telling those apart is exactly what the rest of this module teaches.

In this lesson, you will:

• See why a RAG app is a retriever and a generator in series, and why the final answer alone can’t localize a fault
• Decompose Docent into retrieve() (pure Python) and the Claude generation step
• Map every RAG failure into four quadrants: good/bad retrieval crossed with good/bad generation
• Reproduce the Free-plan retrieval bug deterministically, label its quadrant, and preview the metrics that catch it

Two systems, one answer#

Here is the pipeline in one line: a question goes in, the retriever selects a small set of doc pages, those pages become the context, the generator reads the context and writes an answer, and the answer comes out. Docent’s instruction to the generator is strict — answer using only the context; if it’s not there, say “I don’t have that in the docs.” That instruction is what makes the two stages so tightly coupled: the generator can only be as good as the context the retriever hands it.

Now think about what a wrong answer tells you. If Docent gives you a bad answer, one of two things happened:

• The retriever failed. It fetched the wrong pages, so the answer the user wanted was never in the context. The generator then did exactly what it was told with what it had — which may look like a refusal or a confident answer about the wrong thing.
• The generator failed. The retriever fetched the right pages, the answer was sitting in the context, and the model still got it wrong — misread a number, invented a claim, or refused anyway.

These are completely different bugs with completely different fixes. A retrieval failure is fixed by improving search (better ranking, more pages, embeddings). A generation failure is fixed by improving the prompt or the model. Reading the final answer cannot distinguish them — a wrong answer is a wrong answer. To localize the fault you have to open up the pipeline and score each stage on its own. That is the core idea of RAG evaluation, and everything in this module builds on it.

Decomposing Docent#

Let’s stop talking about the two stages and separate them in code. Here is Docent in full, exactly as you’ve used it — but notice the shape: retrieve() is the entire retriever, and docent_answer() is the generator wrapped around a single call to retrieve(). They are two functions, and we can call them independently.

import warnings
warnings.filterwarnings("ignore")
import anthropic
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

DOCS = [
    {"id": "auth", "title": "Authentication",
     "text": "Meridian authenticates every request with an API key sent in the Authorization header as a bearer token. "
             "Create keys in the dashboard under Settings > API Keys. Keys are either test or live mode; test keys only "
             "touch sandbox data. A key is shown once at creation and cannot be recovered, only rotated."},
    {"id": "ratelimits", "title": "Rate Limits",
     "text": "The Free plan allows 60 requests per minute. The Pro plan allows 600 requests per minute. Exceeding the "
             "limit returns HTTP 429 with a Retry-After header giving the seconds to wait. Rate limits are applied per "
             "API key, not per account."},
    {"id": "plans", "title": "Plans and Pricing",
     "text": "Meridian has three plans. Free costs 0 dollars per month and includes 1 GB of storage. Pro costs 25 dollars "
             "per month and includes 50 GB of storage. Scale costs 199 dollars per month and includes 500 GB of storage. "
             "All plans include unlimited read replicas."},
    {"id": "regions", "title": "Regions",
     "text": "Databases can be created in three regions: us-east, eu-west, and ap-south. The region is fixed at creation "
             "time and cannot be changed later; to move data to another region you must export and re-import into a new "
             "database. Cross-region read replicas are available on the Scale plan only."},
    {"id": "errors", "title": "Error Codes",
     "text": "Meridian uses standard HTTP status codes. 400 means a malformed request, 401 means a missing or invalid API "
             "key, 403 means the key is valid but lacks permission, 404 means the resource does not exist, and 429 means "
             "the rate limit was exceeded. 5xx codes indicate a server-side error and should be retried with backoff."},
    {"id": "backups", "title": "Backups",
     "text": "Automatic daily backups are retained for 7 days on the Free plan, 30 days on Pro, and 90 days on Scale. "
             "Backups run at 03:00 UTC. Point-in-time recovery to any second in the retention window is available on the "
             "Scale plan. Restoring a backup creates a new database and never overwrites the original."},
    {"id": "sdks", "title": "SDKs",
     "text": "Official SDKs are available for Python, JavaScript, and Go. Install the Python SDK with 'pip install meridian'. "
             "The SDK reads the MERIDIAN_API_KEY environment variable by default. All SDKs retry idempotent requests up to "
             "3 times on 5xx errors with exponential backoff."},
    {"id": "deletion", "title": "Deleting Data",
     "text": "Deleting a database is immediate and irreversible once the 24-hour grace period ends. During the grace period "
             "the database is soft-deleted and can be restored from the dashboard. After 24 hours the data and its backups "
             "are permanently purged. Deletion requires a live-mode key with admin permission."},
]

def retrieve(question, docs, k=2):
    # simple keyword-overlap retrieval; good enough for the course, and easy to evaluate
    q = set(question.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())), reverse=True)
    return scored[:k]

def docent_answer(question, docs, model="claude-haiku-4-5"):
    ctx = "\n\n".join(f"[{d['id']}] {d['title']}\n{d['text']}" for d in retrieve(question, docs))
    prompt = (f"Answer the question using ONLY the context. If the context does not contain the answer, say "
              f"\"I don't have that in the docs.\" Be concise.\n\nContext:\n{ctx}\n\nQuestion: {question}")
    msg = client.messages.create(model=model, max_tokens=200,
                                 messages=[{"role": "user", "content": prompt}])
    return msg.content[0].text

The key thing to internalize: retrieve() calls no model. It’s plain Python — a keyword-overlap score and a sort. That means it is deterministic: the same question returns the same doc pages every single time. The generator, docent_answer(), is where the non-determinism lives, because it samples from Claude. When we evaluate RAG we exploit this split constantly — a stage we can reproduce exactly (retrieval) is far easier to diagnose than one that varies (generation), and it’s often where the real bug hides.

Look at retrieval alone#

Before we look at a single answer, let’s look at only what the retriever selects. We’ll take three golden questions with a known correct source page and print the doc ids retrieve() returns, marking whether the page that actually contains the answer made the cut.

questions = {
    "How much does the Pro plan cost per month?": "plans",
    "What does Meridian charge for the Free plan?": "plans",
    "What does a 401 error mean?": "errors",
}

for q, needed in questions.items():
    got = [d["id"] for d in retrieve(q, DOCS)]
    verdict = "HIT " if needed in got else "MISS"
    print(f"[{verdict}] retrieved={got}  needed={needed!r}")
    print(f"        Q: {q}")

[HIT ] retrieved=['ratelimits', 'plans']  needed='plans'
        Q: How much does the Pro plan cost per month?
[MISS] retrieved=['errors', 'backups']  needed='plans'
        Q: What does Meridian charge for the Free plan?
[HIT ] retrieved=['errors', 'auth']  needed='errors'
        Q: What does a 401 error mean?

Read the middle row. For “What does Meridian charge for the Free plan?” the retriever returned ['errors', 'backups'] — and the plans page, the only page that states the Free plan’s price, is nowhere in there. The generator will never see the answer, no matter how good it is. Compare that to the Pro-plan question just above it, phrased almost identically, which does pull plans. The retriever succeeds on one and fails on the other, and we haven’t called Claude even once to know that.

Why does it miss? The retriever scores each page by how many words it shares with the question. The Free-plan question contains “free” and “plan” — but the word “plan” appears in the title of plans and in the text of ratelimits, backups, and regions, while “free” appears in ratelimits (“The Free plan allows 60…”) and backups (“7 days on the Free plan”) too. The plans page happens to share fewer of this question’s exact words than errors and backups do, so it loses the ranking. It’s a brittle keyword match, and it’s wrong in a way that’s totally invisible from the answer.

Because retrieve() is pure Python, this isn’t a fluke of one run. Let’s prove it’s deterministic by running the identical retrieval a second time:

q = "What does Meridian charge for the Free plan?"
for run in (1, 2):
    print(f"run {run}: retrieved =", [d["id"] for d in retrieve(q, DOCS)], "| needed = plans")

run 1: retrieved = ['errors', 'backups'] | needed = plans
run 2: retrieved = ['errors', 'backups'] | needed = plans

Byte-for-byte identical. This bug fires on every request, for every user, forever, until someone fixes the retriever. No amount of prompt-tuning on the generator will touch it.

Now watch it become a false refusal#

We’ve established that the retriever hands the generator the wrong context. Let’s see what the generator does with it. We’ll ask Docent the two questions — the failing Free-plan one and a working 401 one — and print both the retrieved ids and the actual answer, so the cause and effect sit side by side.

for q in ["What does Meridian charge for the Free plan?", "What does a 401 error mean?"]:
    print("Q:", q)
    print("retrieved:", [d["id"] for d in retrieve(q, DOCS)])
    print("A:", docent_answer(q, DOCS))
    print("-" * 60)

Q: What does Meridian charge for the Free plan?
retrieved: ['errors', 'backups']
A: I don't have that in the docs.
------------------------------------------------------------
Q: What does a 401 error mean?
retrieved: ['errors', 'auth']
A: A 401 error means a missing or invalid API key.
------------------------------------------------------------

There it is. Docent said “I don’t have that in the docs” to a question its own documentation answers plainly. And here is the crucial point for evaluation: the generator did nothing wrong. It was handed errors and backups, neither of which mentions the Free plan’s price, and it followed its instruction exactly — answer only from the context, otherwise refuse. Given that context, the refusal is the correct behavior. The model is faithful; the fault is upstream, in retrieval.

This is why reading the answer can’t localize the bug. That refusal is word-for-word what Docent should say to a genuinely out-of-scope question (like “does Meridian support MongoDB aggregation pipelines?”). One is a bug and one is correct behavior, and they produce identical text. The only way to tell them apart is to look at what was retrieved: for the MongoDB question, no page holds the answer and refusing is right; for the Free-plan question, the plans page holds the answer but never made it into the context, so refusing is wrong. Same answer, opposite verdict — decided entirely by the retrieval stage.

(That answer came from one real run against claude-haiku-4-5. The model samples its wording, so a re-run might phrase the refusal slightly differently, but it will still refuse — because the context it’s given genuinely doesn’t contain the price. The retrieval line above, being pure Python, is identical on every run.)

Labeling the quadrant#

With retrieval and generation separated, every RAG outcome falls into one of four boxes — the grid in the figure. Cross “did the retriever fetch the right context?” with “did the generator produce the right answer given its context?”:

• Good retrieval + good generation — the ideal. Right pages in, right answer out. The 401 question above lives here: it retrieved ['errors', 'auth'] (the errors page has the answer) and Claude read it correctly.
• Good retrieval + bad generation — a GENERATION bug. The right pages were in the context, but the model misread a number, invented a claim the context doesn’t support, or refused anyway. The fix is in the prompt or the model. Faithfulness (Lesson 3) is built to catch this.
• Bad retrieval + good generation — a RETRIEVAL bug. The model faithfully answered from the context it was given, but that context was wrong or missing. This is exactly the Free-plan case: ['errors', 'backups'] retrieved, no plans, and a correct-given-the-context refusal that is nonetheless the wrong answer to the user. The fix is in the retriever, not the prompt. Retrieval metrics (Lesson 2) catch this.
• Bad retrieval + bad generation — both broken. The context was wrong and the model added its own error on top. When you see this, fix retrieval first, then re-measure — often the “generation” error disappears once the model finally gets the right context.

So the Free-plan failure is squarely a bad-retrieval + good-generation case: a retrieval bug. If you had only read the answer, you’d have been tempted to “fix” it by rewriting the generator’s prompt — chasing a phantom, because the generator was never the problem. Localizing the fault to the right quadrant is the difference between fixing your search and fixing your prompt, and it’s the practical payoff of evaluating the two stages separately.

Practice Exercises#

Exercise 1: Build a retrieval-only scoreboard#

Using the golden set from the brief (each question has a known source doc id), write a loop that calls retrieve() on each question and prints, for each, the retrieved ids and whether the needed page was among them — no Claude calls at all. Count how many questions retrieved their source page. This is a first, crude retrieval metric; you’ll formalize it in Lesson 2.

Exercise 2: Tell a correct refusal from a broken one#

Docent refuses both the Free-plan question (a bug) and the MongoDB question (correct). Write code that, for each, prints the retrieved ids and then decides — by checking whether the question’s known source page is in the retrieved set — whether the refusal is a retrieval failure or correct out-of-scope behavior. For the MongoDB probe the correct source is None (no page answers it).

Exercise 3: Move the bug to a different quadrant#

The Free-plan failure is a retrieval bug. Change one thing so the plans page is actually retrieved — for example, raise k in retrieve() to 3, or 4 — and re-print the retrieved ids for the Free-plan question. Once plans is in the context, ask Docent again and note which quadrant you’re now in. Did fixing retrieval fix the answer?

Summary#

Docent is not one system but two in series: a deterministic keyword retrieve() that selects doc pages, and a Claude generator that answers using only those pages. Because they’re chained, a wrong final answer can come from either — and reading the answer alone can’t tell you which. You reproduced the Module 1 bug in its two parts: retrieve() returns ['errors', 'backups'] for “What does Meridian charge for the Free plan?” — never the plans page — every single run, and given that wrong context the generator faithfully refuses with “I don’t have that in the docs.” That refusal is correct given its context and wrong for the user, which is why it’s a retrieval bug, not a generation one, even though it looks like the model’s fault. Placed on the four-quadrant grid it’s a clean bad-retrieval + good-generation case, and localizing it there — rather than blaming the prompt — is what evaluating the two stages separately buys you.

Key Concepts#

• RAG is two systems in series — a retriever selects context, a generator answers from it; the generator can only be as good as the context it receives.
• The final answer can’t localize a fault — a wrong answer is consistent with a retrieval bug or a generation bug; you must score each stage to tell them apart.
• Deterministic retrieval — retrieve() calls no model, so it returns the same pages every run; the generator is the non-deterministic half. Debug the reproducible stage first.
• The four quadrants — good/bad retrieval crossed with good/bad generation: ideal, generation bug, retrieval bug, and both-broken, each with a different fix.
• False refusal as a retrieval bug — “I don’t have that in the docs” is correct when no page answers the question but wrong when the answer exists and simply wasn’t retrieved; the retrieved ids, not the text, decide which.

Why This Matters#

When a RAG app gives a bad answer in production, the first and most expensive mistake teams make is fixing the wrong stage — rewriting the generator’s prompt for hours when the retriever was quietly feeding it garbage the whole time. The Free-plan refusal is the canonical example: it reads like the model being unhelpful, but the model was flawless and the search was broken. Being able to say “this is a retrieval failure, not a generation one” turns a vague “the bot is bad” complaint into a specific, fixable engineering target — and it’s only possible if you evaluate the two halves separately. That separation is the foundation of this entire module, and it’s the difference between guessing and diagnosing.

Continue Building Your Skills#

You’ve split Docent into a retriever and a generator and watched a single bug — the wrong two pages retrieved — masquerade as a generation failure. Knowing that the retriever can fail is the first step; the next is measuring how well it does its job, so you can catch these misses before a user does. In Lesson 2 you’ll put numbers on retrieval: precision and recall tell you whether the right pages came back, and MRR and NDCG tell you whether they came back near the top of the list. With those metrics in hand, the Free-plan miss stops being an anecdote you stumbled on and becomes a score you can track, compare, and drive down.