Lesson 3 - Faithfulness & Groundedness

Welcome to Faithfulness & Groundedness#

The last two lessons scored the retriever: did the right Meridian pages come back for a question? This one crosses over to the generator. Suppose retrieval did its job and handed Docent – our assistant for the Meridian serverless database docs – exactly the page it needs. Docent can still fail, and it fails in a peculiarly dangerous way: it can write an answer that reads beautifully, cites the right numbers, and quietly slips in a fact the context never contained. A 30-day free trial. A 99.99 percent uptime guarantee. A retention window that sounds plausible. None of it in the docs. For a documentation assistant, that is the worst outcome – a confident, fluent invention is more harmful than an honest “I don’t know,” because a developer will act on it.

The metric that catches this is faithfulness, also called groundedness: is every claim the answer makes actually supported by the retrieved context? Not “is it true in the world,” not “does it match a reference” – those are correctness, which we measured earlier and will contrast with here. Faithfulness asks only whether the answer stayed inside the four corners of what it was given. An answer that adds anything the context didn’t say is unfaithful, no matter how correct that addition happens to be.

Faithfulness is measurable, and the standard recipe is refreshingly concrete: decompose the answer into small, atomic claims, then check each claim against the context and count how many are entailed. This lesson builds that checker for Docent – live, against claude-haiku-4-5 – and watches it flag a fabricated free trial while a grounded answer scores a clean 1.0.

In this lesson, you will:

• Define faithfulness / groundedness as the fraction of an answer’s atomic claims that are entailed by the retrieved context
• Build a live checker that prompts claude-haiku-4-5 to extract claims and label each SUPPORTED / NOT_SUPPORTED, returning JSON
• Score a grounded Docent answer (~1.0) and one that invents facts (watch it drop and see the flagged claims)
• Separate faithfulness from correctness: an answer can be faithful but wrong, or correct but unfaithful
• Keep the arithmetic deterministic – compute the faithfulness ratio from a fixed claims list and get the identical number every run

What faithfulness measures#

Start with the definition, because the whole method falls out of it. An answer is a bundle of assertions. Break it into its smallest self-contained factual units – atomic claims – and each claim is either entailed by the retrieved context (the context states or clearly implies it) or it isn’t. Faithfulness is just the fraction that are:

A faithfulness of 1.0 means every claim traces back to the context. A faithfulness of 0.5 means half the answer is ungrounded – invented, or drawn from the model’s parametric memory rather than the page it was given. The score lives in \([0, 1]\), and for a docs assistant you want it pinned near 1.0.

Two design decisions make this work. First, decomposition: you don’t grade the answer as one blob, because a single “is this faithful?” verdict throws away which part was invented and blurs a mostly-grounded answer with one bad clause into the same bucket as a total fabrication. Splitting into claims localizes the failure. Second, the check is an entailment judgement – “does this text support this claim?” – which is exactly the kind of fuzzy, semantic call an LLM judge does well and a string match does not. So the checker is itself a small LLM-as-judge, reusing everything from Module 4: a tight prompt, structured JSON out, and honesty about run-to-run wobble.

The deterministic core: counting claims#

Before any model is involved, the scoring itself is pure arithmetic: given a list of claims each labeled SUPPORTED or NOT_SUPPORTED, faithfulness is supported-over-total. Isolating this in its own function keeps the reproducible part reproducible – the number never wobbles, only the labeling that feeds it does.

def faithfulness_ratio(claims):
    """Fraction of atomic claims labeled SUPPORTED. Pure arithmetic, no model."""
    if not claims:
        return 0.0
    supported = sum(1 for c in claims if c["verdict"] == "SUPPORTED")
    return supported / len(claims)

LABELED = [
    {"claim": "The Pro plan allows 600 requests per minute.", "verdict": "SUPPORTED"},
    {"claim": "Rate limits are applied per API key.",          "verdict": "SUPPORTED"},
    {"claim": "The Pro plan costs 30 dollars per month.",       "verdict": "NOT_SUPPORTED"},
]

score = faithfulness_ratio(LABELED)
supported = sum(1 for c in LABELED if c["verdict"] == "SUPPORTED")
print(f"claims: {len(LABELED)}  supported: {supported}  faithfulness: {score:.3f}")
print("run again:", f"{faithfulness_ratio(LABELED):.3f}")

claims: 3  supported: 2  faithfulness: 0.667
run again: 0.667

Two supported claims out of three is 0.667, and it prints the same on the second call because no model touches it – given fixed labels, the ratio is fixed. That is the anchor: whatever variance the LLM introduces when it produces those labels, the moment they exist the score is deterministic. Now we build the part that produces them.

A live faithfulness checker#

The checker takes an answer and the context it should be grounded in, and asks claude-haiku-4-5 to do two jobs in one call: split the answer into atomic claims, and label each one against the context. We demand a single JSON object back, then reuse the fence-stripping JSON extractor from Module 3.

First, the Meridian corpus and retrieval from the running example, so the context we check against is the real retrieved context, not a hand-picked one:

import warnings; warnings.filterwarnings("ignore")
import json, re
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

DOCS = [
    {"id": "auth", "title": "Authentication",
     "text": "Meridian authenticates every request with an API key sent in the Authorization header as a bearer token. "
             "Create keys in the dashboard under Settings > API Keys. Keys are either test or live mode; test keys only "
             "touch sandbox data. A key is shown once at creation and cannot be recovered, only rotated."},
    {"id": "ratelimits", "title": "Rate Limits",
     "text": "The Free plan allows 60 requests per minute. The Pro plan allows 600 requests per minute. Exceeding the "
             "limit returns HTTP 429 with a Retry-After header giving the seconds to wait. Rate limits are applied per "
             "API key, not per account."},
    {"id": "plans", "title": "Plans and Pricing",
     "text": "Meridian has three plans. Free costs 0 dollars per month and includes 1 GB of storage. Pro costs 25 dollars "
             "per month and includes 50 GB of storage. Scale costs 199 dollars per month and includes 500 GB of storage. "
             "All plans include unlimited read replicas."},
    {"id": "regions", "title": "Regions",
     "text": "Databases can be created in three regions: us-east, eu-west, and ap-south. The region is fixed at creation "
             "time and cannot be changed later; to move data to another region you must export and re-import into a new "
             "database. Cross-region read replicas are available on the Scale plan only."},
    {"id": "errors", "title": "Error Codes",
     "text": "Meridian uses standard HTTP status codes. 400 means a malformed request, 401 means a missing or invalid API "
             "key, 403 means the key is valid but lacks permission, 404 means the resource does not exist, and 429 means "
             "the rate limit was exceeded. 5xx codes indicate a server-side error and should be retried with backoff."},
    {"id": "backups", "title": "Backups",
     "text": "Automatic daily backups are retained for 7 days on the Free plan, 30 days on Pro, and 90 days on Scale. "
             "Backups run at 03:00 UTC. Point-in-time recovery to any second in the retention window is available on the "
             "Scale plan. Restoring a backup creates a new database and never overwrites the original."},
    {"id": "sdks", "title": "SDKs",
     "text": "Official SDKs are available for Python, JavaScript, and Go. Install the Python SDK with 'pip install meridian'. "
             "The SDK reads the MERIDIAN_API_KEY environment variable by default. All SDKs retry idempotent requests up to "
             "3 times on 5xx errors with exponential backoff."},
    {"id": "deletion", "title": "Deleting Data",
     "text": "Deleting a database is immediate and irreversible once the 24-hour grace period ends. During the grace period "
             "the database is soft-deleted and can be restored from the dashboard. After 24 hours the data and its backups "
             "are permanently purged. Deletion requires a live-mode key with admin permission."},
]

def retrieve(question, docs, k=2):
    q = set(question.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())), reverse=True)
    return scored[:k]

def build_context(question, docs, k=2):
    return "\n\n".join(f"[{d['id']}] {d['title']}\n{d['text']}" for d in retrieve(question, docs, k))

Now the checker and the JSON plumbing:

def extract_json(text):
    """Pull the first JSON object out of a reply, stripping any markdown fences."""
    fenced = re.search(r"```(?:json)?\s*(.*?)```", text, re.DOTALL)
    candidate = fenced.group(1) if fenced else text
    start, end = candidate.find("{"), candidate.rfind("}")
    if start == -1 or end == -1 or end < start:
        return None
    return candidate[start:end + 1]

def check_faithfulness(answer, context, model="claude-haiku-4-5"):
    """Ask the judge to split the answer into atomic claims and label each against the context."""
    prompt = (
        "You check whether an answer is FAITHFUL to a provided context. "
        "First break the answer into atomic factual claims (each one self-contained fact). "
        "Then label each claim SUPPORTED if the context states or clearly implies it, or "
        "NOT_SUPPORTED if the context does not.\n\n"
        f"Context:\n{context}\n\n"
        f"Answer:\n{answer}\n\n"
        'Respond with one JSON object and nothing else: '
        '{"claims": [{"claim": "...", "verdict": "SUPPORTED"|"NOT_SUPPORTED"}, ...]}.'
    )
    msg = client.messages.create(model=model, max_tokens=400,
                                 messages=[{"role": "user", "content": prompt}])
    return msg.content[0].text

def score_answer(answer, context):
    """Run the live checker and reduce it to (claims, faithfulness)."""
    claims = json.loads(extract_json(check_faithfulness(answer, context)))["claims"]
    return claims, faithfulness_ratio(claims)

Three deliberate choices in that prompt. “First break the answer into atomic factual claims” forces decomposition instead of a single blurry verdict. “SUPPORTED if the context states or clearly implies it” defines entailment loosely enough to allow paraphrase but tightly enough to reject invention. And “one JSON object and nothing else” gives extract_json something clean to parse. Note the example code never sees a key – anthropic.Anthropic() reads it from the environment.

Scoring a faithful and an unfaithful answer#

Now the demonstration the whole lesson is built around. We score two Docent answers. The first is grounded in the retrieved Rate Limits page. The second answers a pricing question but pads the correct price with two facts the docs never mention – a free trial and an uptime SLA – exactly the fluent-fabrication failure a docs assistant must not commit.

# FAITHFUL: every claim comes straight from the retrieved Rate Limits page.
q1 = "How many requests per minute does the Pro plan allow?"
ctx1 = build_context(q1, DOCS)
faithful_answer = "The Pro plan allows 600 requests per minute, and rate limits are applied per API key."

# UNFAITHFUL: the price is right, but the trial and the SLA are invented.
q2 = "How much does the Pro plan cost per month?"
ctx2 = build_context(q2, DOCS)
unfaithful_answer = ("The Pro plan costs 25 dollars per month, includes a 30-day free trial, "
                     "and comes with a 99.99 percent uptime guarantee.")

for label, ans, ctx in [("FAITHFUL", faithful_answer, ctx1),
                         ("UNFAITHFUL", unfaithful_answer, ctx2)]:
    claims, score = score_answer(ans, ctx)
    supported = sum(1 for c in claims if c["verdict"] == "SUPPORTED")
    print(f"[{label}] faithfulness = {score:.2f}  ({supported}/{len(claims)} claims supported)")
    print(json.dumps(claims, indent=2))
    print("-" * 70)

[FAITHFUL] faithfulness = 1.00  (2/2 claims supported)
[
  {
    "claim": "The Pro plan allows 600 requests per minute",
    "verdict": "SUPPORTED"
  },
  {
    "claim": "Rate limits are applied per API key",
    "verdict": "SUPPORTED"
  }
]
----------------------------------------------------------------------
[UNFAITHFUL] faithfulness = 0.33  (1/3 claims supported)
[
  {
    "claim": "The Pro plan costs 25 dollars per month",
    "verdict": "SUPPORTED"
  },
  {
    "claim": "The Pro plan includes a 30-day free trial",
    "verdict": "NOT_SUPPORTED"
  },
  {
    "claim": "The Pro plan comes with a 99.99 percent uptime guarantee",
    "verdict": "NOT_SUPPORTED"
  }
]

That is the metric earning its keep. The faithful answer decomposes into two claims, both traceable to the Rate Limits page, and scores a clean 1.00. The unfaithful answer decomposes into three: the checker correctly marks the 25 dollars claim SUPPORTED – it is on the retrieved Plans page – and flags both the free trial and the uptime SLA as NOT_SUPPORTED, because neither appears anywhere in the context. Faithfulness drops to 0.33, and crucially you can read exactly which claims were invented. A single blob verdict would have told you the answer was “bad”; the decomposition tells you the price is fine and the two garnishes are fabricated – which is the difference between rewriting the prompt and doing nothing.

Faithfulness is not correctness#

It is tempting to treat a high faithfulness score as “the answer is good.” It isn’t – faithfulness and correctness are different axes, and an answer can score high on one and low on the other. Correctness asks “does this match the truth / the reference?” Faithfulness asks “does this match the provided context?” When the context is right, the two usually agree. When it isn’t, they come apart – and that gap is exactly what a RAG evaluation needs to see. Here are three Docent answers run through the live checker, with their correctness noted alongside:

# The REAL Meridian plans page (ground truth for correctness).
real_ctx = ("[plans] Plans and Pricing\nMeridian has three plans. Free costs 0 dollars per month and includes 1 GB "
            "of storage. Pro costs 25 dollars per month and includes 50 GB of storage. Scale costs 199 dollars per "
            "month and includes 500 GB of storage. All plans include unlimited read replicas.")

# A deliberately WRONG context, as if retrieval fed Docent a stale or corrupted page.
wrong_ctx = "[plans] Plans and Pricing\nMeridian's Pro plan costs 40 dollars per month and includes 50 GB of storage."

CASES = [
    dict(label="correct + faithful",
         answer="The Pro plan costs 25 dollars per month.",
         context=real_ctx, correct=True),
    dict(label="correct but UNFAITHFUL",
         answer="The Pro plan costs 25 dollars per month, a price locked in since the 2019 launch.",
         context=real_ctx, correct=True),
    dict(label="faithful but WRONG",
         answer="The Pro plan costs 40 dollars per month.",
         context=wrong_ctx, correct=False),
]

print("Faithfulness is measured against the CONTEXT, not the truth")
print("=" * 66)
for c in CASES:
    claims, f = score_answer(c["answer"], c["context"])
    print(f"\n[{c['label']}]  correct={c['correct']}  faithfulness={f:.2f}")
    for cl in claims:
        print(f"   {cl['verdict']:<14} {cl['claim']}")

Faithfulness is measured against the CONTEXT, not the truth
==================================================================

[correct + faithful]  correct=True  faithfulness=1.00
   SUPPORTED      The Pro plan costs 25 dollars per month.

[correct but UNFAITHFUL]  correct=True  faithfulness=0.50
   SUPPORTED      The Pro plan costs 25 dollars per month
   NOT_SUPPORTED  This price has been locked in since the 2019 launch

[faithful but WRONG]  correct=False  faithfulness=1.00
   SUPPORTED      The Pro plan costs 40 dollars per month.

Read the three rows as the corners of a two-by-two grid. The first is the happy case: right answer, every claim grounded, faithfulness 1.00. The second is correct but unfaithful – the final price is right, so a correctness check passes, yet the checker flags “locked in since the 2019 launch” as NOT_SUPPORTED, dropping faithfulness to 0.50. That invented provenance is the kind of plausible-sounding detail a fluent model adds and a user believes; correctness alone would never catch it. The third is faithful but wrong – retrieval handed Docent a corrupted page saying the Pro plan costs 40 dollars, Docent repeated it exactly, and faithfulness is a perfect 1.00 even though the answer is factually false. That is the metric working as designed: the generator did its job faithfully, and the 1.00 correctly points the blame at retrieval, not generation. If you only measured correctness you’d “fix” the prompt and never touch the broken retriever.

Practice Exercises#

Exercise 1: Score a mixed answer with several claims#

Write a longer Docent answer to “Tell me about Meridian backups” that mixes grounded and invented claims – for example, keep the real retention numbers (7 / 30 / 90 days) but add a fabricated “backups are encrypted with AES-256” that the Backups page never states. Retrieve the real context with build_context, run score_answer, and confirm the faithfulness score lands between 0 and 1 with the invented claim flagged NOT_SUPPORTED.

Exercise 2: Faithful-but-wrong from a real retrieval miss#

The lesson forced a “faithful but wrong” case with a hand-corrupted context. Reproduce it with a real Docent failure instead: find a question where the keyword retrieve pulls the wrong pages (recall the retriever bug from Module 1), build the context, write an answer that faithfully uses whatever wrong page came back, and show that faithfulness is high while the answer is wrong. What does this tell you about where to look when faithfulness is high but users still complain?

Exercise 3: A stricter entailment bar#

Re-run the UNFAITHFUL pricing answer twice: once with the lesson’s prompt (“SUPPORTED if the context states or clearly implies it”) and once after tightening it to “SUPPORTED only if the context explicitly states it; do not credit implications.” Does the stricter bar change any verdicts on the borderline claims, and how many times do you have to run each to feel confident the difference is real and not noise?

Summary#

Faithfulness (groundedness) is the generation-side RAG metric: the fraction of an answer’s atomic claims that are entailed by the retrieved context. You measure it with a two-step recipe – decompose the answer into small self-contained claims, then check each for entailment against the context – and you count supported over total. You built this live for Docent: check_faithfulness prompts claude-haiku-4-5 to split an answer into claims and label each SUPPORTED / NOT_SUPPORTED as JSON, extract_json cleans the reply, and the deterministic faithfulness_ratio reduces the labels to a number that never wobbles. On real runs, a grounded answer scored 1.00 (2 of 2 claims supported) while an answer that padded the correct Pro price with an invented free trial and uptime SLA scored 0.33, with both fabrications flagged so you can see exactly what was invented. Most importantly, you separated faithfulness from correctness: an answer can be correct but unfaithful (the right price plus an invented “locked in since 2019,” faithfulness 0.50) or faithful but wrong (repeating a corrupted context verbatim, faithfulness 1.00 on a false answer) – because faithfulness grades the answer against the context, not the world.

Key Concepts#

• Faithfulness / groundedness – the fraction of an answer’s atomic claims that are entailed by the retrieved context; near 1.0 is the target for a docs assistant.
• Atomic claim decomposition – splitting an answer into its smallest factual units so each can be checked independently and the specific invented claim is localized, not blurred into one verdict.
• Entailment check – a per-claim “does the context support this?” judgement; a fuzzy, semantic call an LLM judge handles well and a string match cannot.
• Graded against context, not truth – faithfulness measures grounding only; a faithful answer built on a wrong context is still faithful, which correctly points blame at retrieval.
• Faithfulness vs. correctness – orthogonal axes: correct-but-unfaithful (right by luck, invented justification) and faithful-but-wrong (bad context) are both real and both diagnostic.
• Deterministic scoring over stochastic labeling – the ratio is fixed once labels exist; only the LLM’s claim extraction and labeling wobble, so average a few runs and don’t over-read one.

Why This Matters#

For any retrieval-augmented assistant, hallucination is the failure that erodes trust fastest, and faithfulness is the metric built to catch it. A correctness check compares the final answer to a reference, but most production questions have no reference sitting ready – what you do have, on every single request, is the context you retrieved, which means faithfulness is measurable online, in production, without labels. That makes it one of the few RAG quality signals you can compute continuously: decompose each live answer, check its claims against the context it was actually given, and alarm when the supported fraction dips. Just as important is what a faithfulness score localizes. A low score says the generator is inventing – tighten the prompt, lower the temperature, or add a “say you don’t know” instruction. A high score paired with wrong answers says the generator is doing its job on bad inputs – the problem is upstream in retrieval. That fork is the entire diagnostic value of evaluating a RAG system as two systems, and it is exactly what the next lesson (answer relevance and context quality) and this module’s guided project build on to tell you not just that Docent failed, but where.

Continue Building Your Skills#

You can now measure whether Docent stayed inside the context it was given: decompose an answer into atomic claims, ask claude-haiku-4-5 to label each SUPPORTED or NOT_SUPPORTED against the retrieved Meridian page, and reduce the labels to a faithfulness score that flags the invented free trial while a grounded answer holds a clean 1.0. You also saw the crucial subtlety – faithfulness grades against the context, not the truth – which is why it can diverge from correctness and why that divergence tells you whether to fix your prompt or your retriever. But faithfulness has a blind spot of its own: an answer can be perfectly grounded and still fail to answer the question, or be grounded in a context that never should have been retrieved. The next lesson measures answer relevance – does the response actually address what was asked – and context quality – was the retrieved material worth grounding in at all – closing the loop from “the generator invented nothing” to “the whole pipeline gave the user what they needed.”