Lesson 4 - Answer Relevance & Context Quality

Welcome to Answer Relevance & Context Quality#

By now you can measure two of the ways Docent – our assistant for the Meridian serverless database docs – goes wrong. Lesson 2 gave you retrieval metrics: did the right pages come back at all? Lesson 3 gave you faithfulness: given the pages, did the generator stick to them or invent claims? Between them they catch a lot. But they leave two gaps, and both are the kind of failure that slips past a demo and lands in front of a user.

The first gap is on the generation side. An answer can be perfectly faithful – every sentence traceable to the retrieved context – and still never answer the question. “Rate limits are important; always check the docs and monitor your usage” invents nothing, contradicts nothing, and is completely useless to someone who asked how many requests per minute the Pro plan allows. Faithfulness scores that answer highly. You need a separate signal that asks the blunt question faithfulness doesn’t: did this actually address what was asked? That’s answer relevance.

The second gap is on the retrieval side, and it’s finer-grained than Lesson 2’s precision-at-k. Two new questions: of the chunks we pulled, how many were actually useful (or did we drag in junk alongside the good page)? And did we retrieve all the context the answer needed, or silently miss a page? Those are context precision and context recall. Together with faithfulness and answer relevance they form a 2×2 – the “RAGAS quartet” – that doesn’t just tell you Docent gave a bad answer, but which half of the pipeline to go fix.

In this lesson, you will:

• See why a faithful answer can still fail, and measure answer relevance with a live claude-haiku-4-5 judge on on-topic, evasive, and off-topic answers
• Compute context precision (useful chunks / retrieved chunks) and context recall (needed chunks retrieved / needed chunks) deterministically from doc ids and the golden set
• Reconnect to Docent’s keyword-retriever bug through the Free-plan question, where context recall is zero because retrieval missed the plans page
• Assemble a per-question RAG report of all four signals and read it as a map: which quadrant dropped tells you whether to fix retrieval or the prompt

The generation gap: faithful but off-topic#

Faithfulness answers “is every claim supported by the context?” Answer relevance answers a different question entirely: “does the answer respond to what the user asked?” These come apart more often than you’d think, because they measure different things – one looks backward from the answer to the context, the other looks sideways from the answer to the question.

Consider three answers to “How many requests per minute does the Pro plan allow?”:

• On-topic: “The Pro plan allows 600 requests per minute.” Directly answers it.
• Evasive: “Rate limits are an important part of using any API responsibly. You should always check the documentation and monitor your usage carefully.” Every word is true and grounded, and it addresses nothing.
• Off-topic: an answer about backup retention when the question was about regions. Coherent, possibly faithful to some context, utterly beside the point.

A faithfulness judge waves the evasive answer through – it makes no unsupported claim. Only a relevance judge catches it. Because “does this address the question?” is a judgement, not a string match, we measure it the same way we measured faithfulness in Lesson 3: an LLM judge that reads the question and the answer and scores how well the answer responds, deliberately ignoring correctness so the two signals stay independent.

import warnings, json, re
warnings.filterwarnings("ignore")
import anthropic
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

def relevance_score(question, answer, model="claude-haiku-4-5"):
    """Judge how well an answer ADDRESSES the question, ignoring correctness."""
    prompt = (
        "You judge ANSWER RELEVANCE only: does the answer respond to what the question "
        "actually asked? Ignore whether it is factually correct -- an off-topic or evasive "
        "answer is low even if true; a direct answer is high even if you can't verify it.\n\n"
        f"Question: {question}\n"
        f"Answer: {answer}\n\n"
        'Respond with a single JSON object and nothing else: '
        '{"rationale": "<one sentence, written first>", "relevance": <float 0.0 to 1.0>}'
    )
    msg = client.messages.create(model=model, max_tokens=180,
                                 messages=[{"role": "user", "content": prompt}])
    text = msg.content[0].text
    m = re.search(r"\{.*\}", text, re.DOTALL)
    obj = json.loads(m.group(0))
    return float(obj["relevance"]), obj["rationale"]

Two design choices carry over from the judge lessons. The prompt says “ignore whether it is factually correct” so relevance doesn’t quietly become a correctness score – an answer can be dead wrong yet perfectly on-topic, and we want to see that. And it asks for the rationale before the number, so the score falls out of reasoning rather than a snap guess. Now run it live on the three answers:

CASES = [
    dict(label="on-topic",
         question="How many requests per minute does the Pro plan allow?",
         answer="The Pro plan allows 600 requests per minute."),
    dict(label="evasive",
         question="How many requests per minute does the Pro plan allow?",
         answer="Rate limits are an important part of using any API responsibly. "
                "You should always check the documentation and monitor your usage carefully."),
    dict(label="off-topic",
         question="Can you change a database's region after it is created?",
         answer="Meridian offers automatic daily backups that are retained for up to 90 days on the Scale plan."),
]

print("Live answer-relevance judging (claude-haiku-4-5)")
print("=" * 74)
for c in CASES:
    score, why = relevance_score(c["question"], c["answer"])
    print(f"\n[{c['label']}] relevance = {score:.2f}")
    print(f"  Q: {c['question']}")
    print(f"  A: {c['answer'][:70]}")
    print(f"  why: {why}")

Live answer-relevance judging (claude-haiku-4-5)
==========================================================================

[on-topic] relevance = 0.95
  Q: How many requests per minute does the Pro plan allow?
  A: The Pro plan allows 600 requests per minute.
  why: The answer directly addresses the specific question about the Pro plan's request rate limit by providing a concrete number.

[evasive] relevance = 0.05
  Q: How many requests per minute does the Pro plan allow?
  A: Rate limits are an important part of using any API responsibly. You sh
  why: The answer provides general advice about rate limits but completely fails to specify the actual number of requests per minute allowed by the Pro plan, which is what was directly asked.

[off-topic] relevance = 0.00
  Q: Can you change a database's region after it is created?
  A: Meridian offers automatic daily backups that are retained for up to 90
  why: The answer discusses backup retention policies rather than addressing whether a database's region can be changed after creation, making it completely off-topic.

The judge cleanly separates the three: the direct answer scores 0.95, the evasive non-answer 0.05, the off-topic one 0.00 – and the evasive case is the one that matters, because it’s the failure faithfulness would have missed. Note that the evasive answer is entirely true and grounded; relevance is what exposes it.

The retrieval gap: precision and recall of the context#

Answer relevance and faithfulness both look at the answer. But if retrieval handed the generator the wrong pages, no amount of faithful, on-topic generation can save the answer – it will be faithfully, relevantly wrong. So we also need to score the context itself, and there are two distinct things to score.

Context precision asks: of the chunks we retrieved, how many were actually relevant? If Docent pulls two pages and only one is useful, precision is 0.5 – we dragged a junk page along with the good one. High precision means a clean context window; low precision means the generator has to work around noise (and pays for the extra tokens).

Context recall asks the opposite: of the chunks the answer needed, how many did we actually retrieve? If the one page that holds the answer never came back, recall is 0 – and the answer is doomed before the model runs. This is Lesson 2’s recall, but framed at the content level: not “did we rank the gold document highly” but “is the information needed to answer here at all?”

Both are simple set arithmetic once you have two things: the ids of the chunks retrieved and the ids of the chunks that are genuinely relevant (from your golden set). No model needed, so these are fully reproducible:

def context_precision(retrieved_ids, relevant_ids):
    """Of the chunks we retrieved, what fraction were actually relevant?"""
    if not retrieved_ids:
        return 0.0
    hits = sum(1 for d in retrieved_ids if d in relevant_ids)
    return hits / len(retrieved_ids)

def context_recall(retrieved_ids, relevant_ids):
    """Of the chunks we NEEDED, what fraction did we actually retrieve?"""
    if not relevant_ids:
        return 1.0
    hits = sum(1 for d in relevant_ids if d in retrieved_ids)
    return hits / len(relevant_ids)

Now run them over three golden questions, using the same keyword retrieve Docent has used all along (it scores each page by word overlap with the question and returns the top k=2). Each golden question carries the id of the page that actually answers it:

DOCS = [
    {"id": "auth", "title": "Authentication",
     "text": "Meridian authenticates every request with an API key sent in the Authorization header as a bearer token. "
             "Create keys in the dashboard under Settings > API Keys. Keys are either test or live mode; test keys only "
             "touch sandbox data. A key is shown once at creation and cannot be recovered, only rotated."},
    {"id": "ratelimits", "title": "Rate Limits",
     "text": "The Free plan allows 60 requests per minute. The Pro plan allows 600 requests per minute. Exceeding the "
             "limit returns HTTP 429 with a Retry-After header giving the seconds to wait. Rate limits are applied per "
             "API key, not per account."},
    {"id": "plans", "title": "Plans and Pricing",
     "text": "Meridian has three plans. Free costs 0 dollars per month and includes 1 GB of storage. Pro costs 25 dollars "
             "per month and includes 50 GB of storage. Scale costs 199 dollars per month and includes 500 GB of storage. "
             "All plans include unlimited read replicas."},
    {"id": "regions", "title": "Regions",
     "text": "Databases can be created in three regions: us-east, eu-west, and ap-south. The region is fixed at creation "
             "time and cannot be changed later; to move data to another region you must export and re-import into a new "
             "database. Cross-region read replicas are available on the Scale plan only."},
    {"id": "errors", "title": "Error Codes",
     "text": "Meridian uses standard HTTP status codes. 400 means a malformed request, 401 means a missing or invalid API "
             "key, 403 means the key is valid but lacks permission, 404 means the resource does not exist, and 429 means "
             "the rate limit was exceeded. 5xx codes indicate a server-side error and should be retried with backoff."},
    {"id": "backups", "title": "Backups",
     "text": "Automatic daily backups are retained for 7 days on the Free plan, 30 days on Pro, and 90 days on Scale. "
             "Backups run at 03:00 UTC. Point-in-time recovery to any second in the retention window is available on the "
             "Scale plan. Restoring a backup creates a new database and never overwrites the original."},
    {"id": "sdks", "title": "SDKs",
     "text": "Official SDKs are available for Python, JavaScript, and Go. Install the Python SDK with 'pip install meridian'. "
             "The SDK reads the MERIDIAN_API_KEY environment variable by default. All SDKs retry idempotent requests up to "
             "3 times on 5xx errors with exponential backoff."},
    {"id": "deletion", "title": "Deleting Data",
     "text": "Deleting a database is immediate and irreversible once the 24-hour grace period ends. During the grace period "
             "the database is soft-deleted and can be restored from the dashboard. After 24 hours the data and its backups "
             "are permanently purged. Deletion requires a live-mode key with admin permission."},
]

def retrieve(question, docs, k=2):
    q = set(question.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())), reverse=True)
    return scored[:k]

GOLD = [
    dict(q="How many requests per minute does the Pro plan allow?", relevant={"ratelimits"}),
    dict(q="Can you change a database's region after it is created?", relevant={"regions"}),
    dict(q="What does Meridian charge for the Free plan?", relevant={"plans"}),
]

print("Deterministic context precision / recall  (retrieve k=2)")
print("=" * 78)
print(f"{'question':<50}{'retrieved ids':<20}{'prec':<7}{'recall'}")
print("-" * 78)
for g in GOLD:
    ids = [d["id"] for d in retrieve(g["q"], DOCS, k=2)]
    p = context_precision(ids, g["relevant"])
    r = context_recall(ids, g["relevant"])
    print(f"{g['q'][:48]:<50}{','.join(ids):<20}{p:<7.2f}{r:.2f}")

Deterministic context precision / recall  (retrieve k=2)
==============================================================================
question                                          retrieved ids       prec   recall
------------------------------------------------------------------------------
How many requests per minute does the Pro plan a  ratelimits,plans    0.50   1.00
Can you change a database's region after it is c  regions,deletion    0.50   1.00
What does Meridian charge for the Free plan?      errors,backups      0.00   0.00

Read the three rows. The first two questions have recall 1.00 – the page that answers them (ratelimits, regions) did come back – but precision 0.50, because k=2 forced a second, useless page along for the ride (plans and deletion respectively). That second page is the junk context precision is built to flag.

The third row is the one to sit with. Asked what Meridian charges for the Free plan, the keyword retriever pulls errors and backups – and not plans, the page that actually holds the pricing. Why? The word “Free” appears in ratelimits (“the Free plan allows 60…”) and backups (“7 days on the Free plan”), and “charge”/“plan” overlap noisily elsewhere, so plans never makes the top 2. Precision is 0.00 (nothing useful retrieved) and, more damning, recall is 0.00: the answer is impossible before the generator even starts. This is Docent’s keyword-retriever bug from Module 1, now caught by a metric instead of by luck. Re-run the block and every number is identical – there’s no model in this loop.

The quartet together: a per-question RAG report#

The four signals are most useful side by side. Faithfulness and answer relevance judge the generator; context precision and recall judge the retriever. Line them up per question and a bad answer stops being a mystery – the pattern of which signals dropped points straight at the cause.

The report below runs the real Docent (docent_answer, the minimal implementation from Module 1) on each golden question, then scores all four signals: precision and recall deterministically from the doc ids, answer relevance and faithfulness with a live claude-haiku-4-5 judge. (Faithfulness reuses Lesson 3’s idea – is every claim in the answer supported by the retrieved context? – scored the same JSON-returning way as relevance.)

def docent_answer(question, docs, model="claude-haiku-4-5"):
    ctx = "\n\n".join(f"[{d['id']}] {d['title']}\n{d['text']}" for d in retrieve(question, docs))
    prompt = (f"Answer the question using ONLY the context. If the context does not contain the answer, say "
              f"\"I don't have that in the docs.\" Be concise.\n\nContext:\n{ctx}\n\nQuestion: {question}")
    msg = client.messages.create(model=model, max_tokens=200,
                                 messages=[{"role": "user", "content": prompt}])
    return msg.content[0].text

def _judge_float(prompt, key, model="claude-haiku-4-5", max_tokens=180):
    msg = client.messages.create(model=model, max_tokens=max_tokens,
                                 messages=[{"role": "user", "content": prompt}])
    m = re.search(r"\{.*\}", msg.content[0].text, re.DOTALL)
    obj = json.loads(m.group(0))
    return float(obj[key]), obj["rationale"]

def answer_relevance(question, answer):
    prompt = (
        "You judge ANSWER RELEVANCE only: does the answer respond to what the question asked? "
        "Ignore correctness; an evasive or off-topic answer is low even if true.\n\n"
        f"Question: {question}\nAnswer: {answer}\n\n"
        'Respond with one JSON object: {"rationale": "<one sentence first>", "relevance": <float 0.0-1.0>}')
    return _judge_float(prompt, "relevance")

def faithfulness(answer, context):
    prompt = (
        "You judge FAITHFULNESS: is every claim in the answer supported by the context? "
        "A refusal that adds no claims is faithful.\n\n"
        f"Context:\n{context}\n\nAnswer: {answer}\n\n"
        'Respond with one JSON object: {"rationale": "<one sentence first>", "faithfulness": <float 0.0-1.0>}')
    return _judge_float(prompt, "faithfulness")

print("Mini per-question RAG report -- the RAGAS quartet on Docent")
print("=" * 74)
for g in GOLD:
    got = retrieve(g["q"], DOCS, k=2)
    ids = [d["id"] for d in got]
    ctx = "\n\n".join(f"[{d['id']}] {d['title']}\n{d['text']}" for d in got)
    ans = docent_answer(g["q"], DOCS)
    cp = context_precision(ids, g["relevant"])
    cr = context_recall(ids, g["relevant"])
    ar, _ = answer_relevance(g["q"], ans)
    fa, _ = faithfulness(ans, ctx)
    print(f"\nQ: {g['q']}")
    print(f"  answer:  {ans[:72]}")
    print(f"  RETRIEVAL   context_precision={cp:.2f}  context_recall={cr:.2f}  (relevant={sorted(g['relevant'])}, got={ids})")
    print(f"  GENERATION  answer_relevance={ar:.2f}  faithfulness={fa:.2f}")

Mini per-question RAG report -- the RAGAS quartet on Docent
==========================================================================

Q: How many requests per minute does the Pro plan allow?
  answer:  The Pro plan allows 600 requests per minute.
  RETRIEVAL   context_precision=0.50  context_recall=1.00  (relevant=['ratelimits'], got=['ratelimits', 'plans'])
  GENERATION  answer_relevance=0.95  faithfulness=1.00

Q: Can you change a database's region after it is created?
  answer:  No. According to the context, the region is fixed at creation time and c
  RETRIEVAL   context_precision=0.50  context_recall=1.00  (relevant=['regions'], got=['regions', 'deletion'])
  GENERATION  answer_relevance=0.95  faithfulness=1.00

Q: What does Meridian charge for the Free plan?
  answer:  I don't have that in the docs.
  RETRIEVAL   context_precision=0.00  context_recall=0.00  (relevant=['plans'], got=['errors', 'backups'])
  GENERATION  answer_relevance=0.15  faithfulness=1.00

Now read the quartet as a diagnostic map. The first two questions are healthy: recall 1.00 (the answer’s page was retrieved), relevance ~0.95 (Docent answered directly), faithfulness 1.00 (nothing invented). The only blemish is precision 0.50 – a wasted second page, worth tightening but not causing a wrong answer.

The Free-plan row is a failure, and the quartet tells you exactly whose fault it is. Context recall is 0.00 – the plans page never made it into the window – so the retriever failed. Handed only errors and backups, the generator did the right thing: it refused with “I don’t have that in the docs.” That refusal is faithful (1.00 – it invents nothing) but scores low on relevance (0.15 – from the user’s point of view, the question went unanswered). This is the pattern to internalize: a low-relevance, high-faithfulness answer sitting on top of zero recall is not a prompt problem or a hallucination – it’s a retrieval problem, and no amount of prompt-tuning will fix it. You go fix the search.

The deterministic half of this report (precision, recall) is identical on every run. The live half (relevance, faithfulness) will wobble a little – relevance on the refusal might read 0.1 or 0.2 next time, faithfulness stays pinned at or near 1.0 – but the shape of the diagnosis is stable: two healthy rows and one retrieval failure.

Practice Exercises#

Exercise 1: Faithful but off-topic, on purpose#

The lesson claimed an answer can be faithful yet irrelevant. Prove it end to end. Take the Pro-plan question, retrieve its real context, and hand the generator this answer instead: "Rate limits protect the service and are applied per API key, not per account." Score both faithfulness (against the retrieved context) and answer relevance (against the question). You should get high faithfulness and low relevance – the two signals disagreeing on the same answer.

Exercise 2: Raise recall by retrieving more#

Context recall was 0.00 on the Free-plan question because plans didn’t make the top 2. Re-run the deterministic precision/recall block with k=3 and then k=5. At what k does plans finally get retrieved, and what happens to context precision on the questions that already had recall 1.00 as you raise k?

Exercise 3: Add a “healthy” and a “broken” question and read the report#

Extend GOLD with two more golden questions – one you expect the keyword retriever to handle well (recall 1.00) and one you suspect it will fumble (recall 0.00) – and run the full quartet report on all five. Before you run it, predict each question’s four signals from what you know about the retriever, then check yourself.

Summary#

You completed the RAG metric picture – the RAGAS quartet – by adding the two signals that retrieval metrics and faithfulness leave out. On the generation side, answer relevance asks whether the answer actually addresses the question, catching the faithful-but-evasive failure that faithfulness waves through: a live claude-haiku-4-5 judge scored an on-topic answer 0.95, an evasive-but-true one 0.05, and an off-topic one 0.00. On the retrieval side, context precision (useful chunks / retrieved chunks) and context recall (needed chunks retrieved / needed chunks) score the context itself; computed deterministically from doc ids and the golden set, they gave precision 0.50 on the two well-answered questions (a junk page dragged along by k=2) and, crucially, recall 0.00 on the Free-plan question, where Docent’s keyword retriever pulled errors and backups and missed the plans page entirely. Assembling all four into a per-question report turned a bad answer into a diagnosis: the Free-plan row – recall 0.00, relevance 0.15, faithfulness 1.00 – localizes the failure to the retriever, since the generator did the honest thing and refused. Four numbers say what one cannot: a good generator sitting on top of a broken search.

Key Concepts#

• Answer relevance – does the answer respond to what was asked? Scored by an LLM judge that ignores correctness, so it catches a faithful, grounded answer that’s still evasive or off-topic.
• Context precision – of the retrieved chunks, the fraction that were actually relevant; low precision means junk pulled alongside the useful page (and wasted context tokens).
• Context recall – of the chunks the answer needed, the fraction actually retrieved; recall of 0 means the answer is impossible before the generator runs.
• The RAGAS quartet – faithfulness + answer relevance (generator) and context precision + context recall (retriever), a 2×2 that localizes a failure to the half of the pipeline that caused it.
• Deterministic vs live signals – precision and recall are set arithmetic and reproducible; relevance and faithfulness are judge calls that wobble slightly, so trust their ranking and buckets, not the third decimal.
• Refusal is faithful, not relevant – an honest “I don’t have that in the docs” scores high on faithfulness but low on relevance, a signature that points at retrieval rather than the prompt.

Why This Matters#

RAG is two systems in one, and every production RAG incident is really the question “was that the retriever or the generator?” A single end-to-end score can’t answer it – it tells you the answer was bad, not why, so your team argues about whether to re-tune the prompt or rebuild the index. The quartet answers it structurally. When recall drops, you fix search; when faithfulness drops, you fix the prompt or the model; when relevance drops on a faithful answer, the generator is dodging; when precision drops, you’re burning tokens on noise. That’s the difference between a metric you can act on and a metric you can only worry about. It’s also why answer relevance has to be measured separately from faithfulness at all: the single most common way a “grounded” RAG system disappoints users is by being faithfully, honestly beside the point – and you will never see it if faithfulness is the only generation number you watch. With all four signals in hand, you’re ready for the module’s capstone, where you’ll run this quartet across a whole evaluation set and produce a report that diagnoses each failure as retrieval or generation, end to end.

Continue Building Your Skills#

You can now score all four corners of RAG quality on Docent. Answer relevance – a live judge that asks “did this address the question?” and ignores correctness – catches the faithful-but-evasive answer that faithfulness alone lets through, scoring a direct answer 0.95 and a true-but-useless one 0.05. Context precision and context recall, computed deterministically from retrieved doc ids against the golden set, score the context itself: the junk page dragged along by k=2 (precision 0.50) and, sharper, the plans page the keyword retriever silently missed on the Free-plan question (recall 0.00). And the per-question RAG report ties the quartet into a diagnosis rather than a verdict – the Free-plan row’s recall 0.00, relevance 0.15, faithfulness 1.00 says, unambiguously, “good generator, broken retriever, go fix the search.” The next lesson is the module’s guided project: you’ll scale this quartet from three hand-picked questions to a full evaluation set and build the report that, for every failing answer, tells you which half of the pipeline to fix.