How to Evaluate RAG Answers in Python Without an LLM Judge

A retrieval-augmented generation system can return a fluent answer and still fail. It may retrieve the wrong policy, omit an important condition, cite an unrelated record, invent a detail, or answer a question that the knowledge base cannot support. Reading a few convincing examples will not reveal these different errors.

To evaluate RAG answers in Python without an LLM judge, create fixed questions with expected evidence, required answer points, claim-level support labels, and an abstention label. Retrieve documents for every question, compare the saved answer with those labels, and report each check separately so one strong metric cannot hide another failure.

This tutorial builds a small offline evaluator for saved RAG answers. RAG, or retrieval-augmented generation, retrieves relevant text and gives it to a language model as context for an answer. We will test retrieval and answer quality separately, without calling a model API or using another model as a judge.

Prerequisites and Setup

You need Python 3.10 or later and basic knowledge of lists, functions, and tables. A pandas DataFrame is a table with named columns. This lesson uses pandas for the evaluation table, scikit-learn for text retrieval, and Matplotlib for the chart.

Create a virtual environment and activate it. Then install the three packages:

python -m venv .venv
source .venv/bin/activate
python -m pip install pandas scikit-learn matplotlib

On Windows PowerShell, use .venv\Scripts\Activate.ps1 for the activation command.

The executed build used Python 3.13.2, pandas 3.0.3, scikit-learn 1.9.0, and Matplotlib 3.11.0. Download these original teaching datasets and save them beside your Python file:

• workshop policy chunks
• RAG evaluation cases
• claim-to-evidence annotations

All three datasets are synthetic. The workshop, policies, contact address, questions, and candidate answers are fictional. The datasets were created for this lesson and are released under CC0 1.0. You can also inspect the executed per-case results.

Mental Model: Test the Evidence Chain One Link at a Time

A RAG answer passes through a chain:

question -> retrieved chunks -> answer claims -> citations -> user

A chunk is a small piece of a larger document that the retriever can return. A claim is one checkable statement in an answer. For example, “Entry is allowed up to 15 minutes late” is one claim. “Reception can always replay the safety briefing” is another claim, even when both appear in one answer.

We will give every case five scores between 0 and 1:

1. Retrieval recall asks what share of the expected chunks appeared in the top results.
2. Citation precision asks what share of the cited chunks were both retrieved and labeled as expected evidence.
3. Claim support asks what share of annotated answer claims have a human-identified supporting chunk.
4. Key-point coverage asks what share of required facts appear in the answer.
5. Abstention correctness asks whether the system refused only when the knowledge base lacked an answer.

An abstention is a deliberate refusal to answer from insufficient evidence. It is a correct result for a question about lunch when the knowledge base contains no lunch policy.

These checks are intentionally separate. Good retrieval does not prove that the answer used the retrieved text. A valid citation does not prove that every claim is supported. A complete answer can still be wrong. The LLM pipeline testing tutorial tests whether program boundaries handle expected inputs and failures. This lesson measures the saved evidence and answer content inside those boundaries.

Build a Small Evaluation Set You Can Inspect

The fictional knowledge base contains eight workshop policies. Load the CSV and inspect the first three chunks before building a retriever:

import pandas as pd

chunks = pd.read_csv("workshop_policy_chunks.csv")
cases = pd.read_csv("rag_evaluation_cases.csv")
claims = pd.read_csv("rag_claim_annotations.csv")

print(chunks.head(3).to_string(index=False))

The executed data starts like this:

chunk_id                  topic                                                                                                                                                text
     W01  registration deadline                      Registration for a workshop closes 48 hours before its listed start time. Places cannot be held after the deadline.
     W02    cancellation credit                       A cancellation made at least 72 hours before the start receives a full account credit. Later cancellations receive no credit.
     W03 accessibility support Accessibility requests should be sent to [email protected] at least five calendar days before a workshop. A support coordinator replies within two working days.

Each evaluation row stores a question, expected chunk IDs, required answer points, a saved candidate answer, its citations, and the expected abstention decision. Saved answers make this evaluation deterministic: the same file produces the same results without network access or model variation.

Inspect a few columns so the contract is visible:

columns = [
    "case_id", "question", "expected_chunk_ids",
    "required_points", "cited_chunk_ids", "should_abstain",
]
print(cases[columns].head(4).to_string(index=False))

case_id                                                                            question expected_chunk_ids                        required_points cited_chunk_ids  should_abstain
    Q01                                         When does registration close for a workshop?                W01                               48 hours             W01            False
    Q02                               What credit do I get if I cancel four days before?                W02                    full account credit             W02            False
    Q03 How early should I request accessibility support, and when will someone reply?                W03 five calendar days|two working days             W03            False
    Q04                                                   Can I enter 10 minutes late?                W06                             15 minutes             W06            False

The vertical bar separates multiple expected items. Q03 needs both the request deadline and the reply time. Its candidate answer includes only the deadline, so a completeness check should catch the missing reply time.

The ten cases contain deliberate failures. That is useful. An evaluation set containing only perfect answers cannot show whether a metric reaches the intended failure path.

Retrieve the Top Evidence with TF-IDF

We need a visible retriever before evaluating retrieval. TF-IDF gives more weight to words that are useful in one document and less weight to words that occur across many documents. scikit-learn’s current TfidfVectorizer documentation describes it as converting raw documents into a TF-IDF feature matrix.

The next function learns a vocabulary from the chunk text, transforms each question with the same vocabulary, and returns the two most similar chunks:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

def retrieve_top_k(chunks, questions, k=2):
    vectorizer = TfidfVectorizer(
        ngram_range=(1, 2),
        stop_words="english",
    )
    chunk_vectors = vectorizer.fit_transform(chunks["text"])
    question_vectors = vectorizer.transform(questions)
    similarities = cosine_similarity(question_vectors, chunk_vectors)

    return [
        chunks.iloc[row.argsort()[::-1][:k]]["chunk_id"].tolist()
        for row in similarities
    ]

cases["retrieved_chunk_ids"] = [
    "|".join(ids) for ids in retrieve_top_k(chunks, cases["question"])
]

Cosine similarity compares the direction of two numeric vectors. Similar text receives a larger score even when document lengths differ. The scikit-learn pairwise-metrics guide notes that cosine similarity is commonly used for documents represented by TF-IDF vectors.

This lexical retriever is a teaching baseline, not a required production choice. You can replace its results with saved output from a vector database, hybrid search, or another retrieval system. Keep the same evaluator columns.

Measure Retrieval Recall and Citation Precision

First, turn a pipe-separated cell into a Python set. Missing CSV cells become NaN in pandas, so the function must handle them explicitly:

def split_pipe_values(value):
    if pd.isna(value) or not str(value).strip():
        return set()
    return {
        item.strip()
        for item in str(value).split("|")
        if item.strip()
    }

Retrieval recall is the fraction of expected chunks found among the top results. An unanswerable case has no expected chunk, so it receives 1.0 here; abstention checks whether the system handled that absence correctly.

The next helper calculates recall, then applies it to every row:

def fraction_found(required, observed):
    if not required:
        return 1.0
    return len(required & observed) / len(required)

cases["retrieval_recall"] = cases.apply(
    lambda row: fraction_found(
        split_pipe_values(row["expected_chunk_ids"]),
        split_pipe_values(row["retrieved_chunk_ids"]),
    ),
    axis=1,
)

Citation precision uses a stricter intersection. A cited chunk counts as valid only when it was retrieved and labeled as expected evidence for that question. An answerable case with no citation fails; a correct abstention with no citation passes.

The following function implements that policy:

def citation_precision(row):
    cited = split_pipe_values(row["cited_chunk_ids"])
    if not cited:
        return 1.0 if row["should_abstain"] else 0.0

    valid = (
        cited
        & split_pipe_values(row["retrieved_chunk_ids"])
        & split_pipe_values(row["expected_chunk_ids"])
    )
    return len(valid) / len(cited)

cases["citation_precision"] = cases.apply(
    citation_precision,
    axis=1,
)

Q05 retrieves the correct online-link policy, but its saved answer cites the registration policy. Retrieval recall passes while citation precision fails. This is exactly why the two measures should not be merged.

Check Claim Support and Required Answer Points

Automated word overlap is not enough to decide whether a claim is supported. A false sentence can repeat many words from a policy. For this small evaluation, a human reviewer splits candidate answers into claims and records which chunk supports each claim. An empty support field means the knowledge base does not support that claim.

Calculate a Boolean support value for each annotated claim, then average within each case:

claims["supported"] = claims["support_chunk_ids"].map(
    lambda value: bool(split_pipe_values(value))
)

claim_support = (
    claims.groupby("case_id")["supported"]
    .mean()
    .to_dict()
)

cases["claim_support"] = cases.apply(
    lambda row: claim_support.get(
        row["case_id"],
        1.0 if row["candidate_abstains"] else 0.0,
    ),
    axis=1,
)

A case with no annotated claim passes this check only when the candidate abstains. An answerable case without claim annotations receives 0.0, which prevents missing review work from silently passing.

Q04 contains two claims. The 15-minute entry rule is supported, but the sentence about replaying the safety briefing is not. Its claim-support score is therefore 1 divided by 2, or 0.5.

Support does not measure completeness. Q03 has one supported claim but omits the reply time. We handle simple fixed facts with required phrases:

import re

def point_coverage(required, answer):
    points = split_pipe_values(required)
    normalized_answer = re.sub(
        r"\s+", " ", answer.lower()
    ).strip()

    if not points:
        return 1.0
    found = sum(
        point.lower() in normalized_answer
        for point in points
    )
    return found / len(points)

cases["key_point_coverage"] = cases.apply(
    lambda row: point_coverage(
        row["required_points"],
        row["candidate_answer"],
    ),
    axis=1,
)

Literal phrase matching is suitable for stable values such as “24 hours.” It is fragile for open-ended language. If several phrasings are valid, store accepted alternatives or use human review. Do not report this narrow check as general semantic accuracy.

Test Abstention and Combine the Checks

The dataset stores both should_abstain and candidate_abstains. Comparing them catches two opposite errors: answering an unsupported question and refusing an answerable one.

Create the abstention result, then require every metric to equal 1.0 for a full pass:

cases["abstention_correct"] = cases[
    "should_abstain"
].eq(cases["candidate_abstains"]).astype(float)

metric_columns = [
    "retrieval_recall",
    "citation_precision",
    "claim_support",
    "key_point_coverage",
    "abstention_correct",
]

cases["passes_all"] = (
    cases[metric_columns].eq(1.0).all(axis=1)
)

The full pass is a regression rule, not a universal quality standard. For high-impact applications, you may require human approval or stricter thresholds. Keep the individual values so you can identify the broken link.

The executed build printed this summary:

cases=10 passes_all=5
retrieval_recall       1.000
citation_precision     0.800
claim_support          0.750
key_point_coverage     0.850
abstention_correct     0.900
failed_cases=Q03,Q04,Q05,Q07,Q10

All expected evidence appeared in the top two results, so mean retrieval recall is 1.0. That does not make the system successful: only five cases pass every check. Mean citation precision falls to 0.8 because Q05 cites the wrong policy and Q07 cites a policy for an unanswerable parking question. Mean claim support is lowest because Q04, Q07, and Q10 contain unsupported or contradicted claims.

Read the chart across metrics; do not treat the bars as one model score. The perfect retrieval bar says the evidence finder found all expected chunks in the answerable cases. The lower answer-level bars show that the saved answers still need correction.

Print failed rows during development. The case IDs tell you where to inspect the question, answer, expected evidence, and annotations:

failed = cases.loc[
    ~cases["passes_all"],
    ["case_id", *metric_columns],
]
print(failed.to_string(index=False))

case_id  retrieval_recall  citation_precision  claim_support  key_point_coverage  abstention_correct
    Q03               1.0                 1.0            1.0                 0.5                 1.0
    Q04               1.0                 1.0            0.5                 1.0                 1.0
    Q05               1.0                 0.0            1.0                 1.0                 1.0
    Q07               1.0                 0.0            0.0                 1.0                 0.0
    Q10               1.0                 1.0            0.0                 0.0                 1.0

Each row has a different diagnosis. Q03 needs a more complete answer. Q04 must remove an invented sentence. Q05 must cite the retrieved online-link policy. Q07 should abstain. Q10 contradicts the seven-day return rule.

Apply the Evaluator to Your RAG System

Keep evaluation separate from live generation. First, freeze a versioned knowledge base and reviewed cases. Then run your RAG system and save these fields for each case:

• the exact question;
• retrieved chunk IDs in ranked order;
• the complete answer text;
• cited chunk IDs; and
• whether the system abstained.

Join that run with expected evidence, required points, and claim annotations by a stable case_id. Record the retriever version, model name, prompt version, generation settings, knowledge-base version, and run date. These details make comparisons meaningful when one component changes.

Use synthetic cases for clear rules and privacy-safe failure paths. Add governed, de-identified examples from the intended task when your data policy allows it. Split development and final evaluation cases. If you repeatedly change a prompt after reading one case, that case is now development data.

You can also compare two saved runs case by case. The paired-bootstrap prompt tutorial shows how to estimate uncertainty when the same labeled cases are evaluated under two versions.

Common Mistakes and Troubleshooting

One total score hides the failure type. Keep retrieval, citations, claims, completeness, and abstention in separate columns. A weighted average can make an unsafe answer look acceptable.

Retrieved text is treated as relevant evidence. A retriever always returns something when k is greater than zero. Compare returned IDs with reviewed expected evidence. For unanswerable questions, retrieved chunks are candidates, not proof that an answer exists.

Citations are checked only for valid ID syntax. An existing chunk ID may still be unrelated. Require a citation to match reviewed evidence and inspect whether it supports the nearby claim.

Claim labels are generated by the system being tested. Use independent human review for the evaluation labels. For important or ambiguous cases, ask a second reviewer and resolve disagreements before scoring.

Required phrases are mistaken for semantic evaluation. Exact matching can miss correct paraphrases and accept a phrase used in the wrong way. Limit it to stable facts, or define reviewed alternatives. Read failures before changing the rule.

Only answerable questions are included. Add questions that the knowledge base cannot answer. Otherwise, a system that always produces an answer is never tested for safe abstention.

The dataset is too small or too clean. Ten cases are enough to learn the method, not to approve a production system. Add spelling variation, multi-chunk questions, near-duplicate policies, conflicting versions, long contexts, supported languages, and failures observed in the intended application.

Results change after a package or data update. Save package versions, knowledge-base versions, and outputs. Run the evaluator in a fixed environment before comparing releases.

The chart script fails on a server. Select Matplotlib’s Agg backend before importing matplotlib.pyplot. Agg writes the SVG without requiring a desktop display.

Recap

An offline RAG evaluation should follow the evidence chain from question to retrieval to answer. Start with fixed cases and reviewed expectations. Then measure whether the expected chunks were retrieved, citations point to that evidence, each claim is supported, required facts are present, and unsupported questions receive an abstention.

The ten synthetic cases produced perfect retrieval recall but only five full passes. That difference is the main lesson: retrieving the right chunk is necessary, but it does not guarantee a grounded, complete, correctly cited answer.

Use deterministic checks for fast regression testing and human annotations for meaning that simple code cannot judge. If you later add an LLM judge, keep these transparent checks. They give reviewers concrete evidence when the judge score changes or disagrees with a labeled case.