How to Detect Unsupported Numbers in RAG Answers with Python

A handbook says to log a sensor reading every 60 seconds when its battery is low. A generated answer says 45 seconds and cites that handbook. The citation looks reassuring, but the number is not in the cited evidence.

To detect unsupported numbers in RAG answers, evaluate one claim at a time, require it to cite a document that was actually retrieved, and extract numeric facts from both the claim and its cited evidence. Normalize equivalent units such as 1 minute and 60 seconds, then flag the claim if any quantity or date from the claim is absent from the evidence; send claims with no recognized numeric facts to a separate review path.

This tutorial builds that check with standard Python and tests it on 16 saved claims. RAG, or retrieval-augmented generation, retrieves text and gives it to a language model as context. A claim is one statement that can be checked on its own. We will check quantities inside claims, not try to judge every part of natural language.

What You Need and What the Data Represents

You need Python 3.10 or later and basic knowledge of functions, dictionaries, and CSV files. A regular expression, often shortened to regex, is a text pattern used to find items such as 15 seconds or 2026-09-30. Matplotlib is needed only for the results chart.

Create and activate a virtual environment, then install Matplotlib. On macOS or Linux, run:

python -m venv .venv
source .venv/bin/activate
python -m pip install matplotlib

On Windows PowerShell, use .venv\Scripts\Activate.ps1 for the activation command. The executed build used Python 3.13.2 and Matplotlib 3.11.0.

Download the two teaching inputs and save them beside your Python file:

• fictional field-sensor handbook
• saved RAG numeric claims

Both datasets are original synthetic data created for this lesson and released under CC0 1.0. All document IDs, equipment rules, dates, thresholds, claims, and retrieval results are fictional. They do not describe a real sensor or organization. You can also inspect the executed claim results.

Load the files with Python’s standard csv module. The next step keeps the handbook in a dictionary so a citation ID can find its evidence text directly:

import csv


def read_csv(path):
    with open(path, newline="", encoding="utf-8") as handle:
        return list(csv.DictReader(handle))

handbook_rows = read_csv("field_sensor_handbook.csv")
claim_rows = read_csv("rag_numeric_claims.csv")

documents = {
    row["doc_id"]: row["text"]
    for row in handbook_rows
}

for row in handbook_rows[:3]:
    print(row["doc_id"], row["topic"])

The executed code prints these first three IDs and topics:

S01 battery logging
S02 flow calibration
S03 probe storage

The handbook has eight records. The claim file has 16 cases, including correct quantities, altered quantities, missing citations, unknown IDs, and citations that were not part of retrieval. expected_status is a teaching label used to test the checker. It is not produced by a language model.

Understanding Numeric Evidence as Comparable Signatures

The checker follows one claim through four gates:

claim
  -> citation exists
  -> cited document was retrieved
  -> claim quantities can be extracted
  -> every normalized quantity appears in the cited text

The last gate needs a representation that ignores harmless wording differences. The strings 1 minute and 60 seconds differ, but both can become this signature:

category=time_seconds, value=60

A signature is a small, comparable description of a fact. This tutorial uses five measured-quantity categories, plus date strings:

• time, converted to seconds;
• percentage, kept as a percentage value;
• temperature, kept in degrees Celsius;
• flow, kept in liters per minute;
• file size, kept in megabytes; and
• dates from 2000 through 2099, kept as YYYY-MM-DD strings.

This approach is deliberately narrow. It can catch 45 seconds when the citation contains 60 seconds. It cannot decide whether “increase to 60 seconds” and “do not increase to 60 seconds” have opposite meanings. A passing numeric check means only that the quantities are present in the cited evidence.

If you need a wider evaluation across retrieval, citations, completeness, and abstention, see how to evaluate RAG answers without an LLM judge. Here, we keep one guardrail small enough to understand line by line.

Guard Citations Before Comparing Values

Do not compare a claim with arbitrary handbook text. First confirm that the claimed citation is real and that the retriever supplied it for this question. Otherwise, an answer could cite a valid but unrelated document that the model never received.

The CSV stores retrieved IDs with a vertical bar. This helper converts a value such as S01|S03 into a set:

def split_ids(value: str) -> set[str]:
    return {
        item.strip()
        for item in value.split("|")
        if item.strip()
    }

Now apply the citation gates in a fixed order. Each stop path has a different status, which makes the result useful for debugging:

def check_citation(claim, documents):
    citation = claim["cited_doc_id"].strip()
    retrieved = split_ids(claim["retrieved_doc_ids"])

    if not citation:
        return "missing_citation"
    if citation not in documents:
        return "unknown_citation"
    if citation not in retrieved:
        return "citation_not_retrieved"
    return "citation_ok"

These checks answer three different questions. missing_citation means there is no evidence pointer. unknown_citation may indicate an invented or stale ID. citation_not_retrieved means the document exists, but it was not listed among the chunks supplied for that case.

Passing these gates still does not prove support. It only identifies the exact evidence that the quantity comparison should inspect.

Extract and Normalize Quantities

We will represent a normalized fact with a frozen data class. “Frozen” means its fields cannot change after creation, which also lets Python store facts in a set:

from dataclasses import dataclass
from decimal import Decimal

@dataclass(frozen=True, order=True)
class Fact:
    category: str
    value: Decimal | str

    def label(self) -> str:
        return f"{self.category}={self.value}"

Decimal stores values such as 1.5 in decimal form. That avoids unnecessary binary floating-point differences during equality checks. Python’s official decimal documentation explains its exact decimal representation and arithmetic.

Next, define one pattern for date-shaped strings and another for supported number-unit pairs. The date pattern accepts YYYY-MM-DD strings from 2000 through 2099; it does not check whether a date such as 2026-02-31 exists on the calendar. Named groups give the number and unit clear labels:

import re

DATE_PATTERN = re.compile(
    r"\b20\d{2}-(?:0[1-9]|1[0-2])-(?:0[1-9]|[12]\d|3[01])\b"
)

QUANTITY_PATTERN = re.compile(
    r"(?P<value>\d+(?:\.\d+)?)\s*-?\s*"
    r"(?P<unit>"
    r"liters?\s+per\s+minute|l/min|"
    r"degrees?\s+celsius|°\s*c|"
    r"megabytes?|mb|percent|%|"
    r"seconds?|minutes?|hours?|days?|weeks?"
    r")(?=$|\W)",
    re.IGNORECASE,
)

The optional hyphen allows both 15 seconds and 15-second. The official Python re reference documents named groups and finditer(), which we use to visit every match rather than only the first one.

Normalization converts time units to seconds. The same function assigns the other supported units to stable categories:

def canonical_fact(value_text: str, unit_text: str) -> Fact:
    value = Decimal(value_text)
    unit = re.sub(
        r"\s+", " ",
        unit_text.lower().replace("°", "")
    ).strip()

    time_factors = {
        "second": Decimal(1),
        "seconds": Decimal(1),
        "minute": Decimal(60),
        "minutes": Decimal(60),
        "hour": Decimal(3600),
        "hours": Decimal(3600),
        "day": Decimal(86400),
        "days": Decimal(86400),
        "week": Decimal(604800),
        "weeks": Decimal(604800),
    }
    if unit in time_factors:
        return Fact("time_seconds", value * time_factors[unit])
    if unit in {"percent", "%"}:
        return Fact("percent", value)
    if unit in {"degree celsius", "degrees celsius", "c"}:
        return Fact("temperature_c", value)
    if unit in {"liter per minute", "liters per minute", "l/min"}:
        return Fact("flow_l_per_min", value)
    if unit in {"megabyte", "megabytes", "mb"}:
        return Fact("size_mb", value)
    raise ValueError(f"unsupported unit: {unit_text}")

The complete build also accepts short forms such as mins, British spelling such as litres, and hrs. Add synonyms only when you have a labeled case for them. A large permissive pattern can silently extract the wrong text.

Finally, extract all recognized facts. Date-shaped values remain strings because this narrow checker only compares them exactly; measured quantities use Decimal:

def extract_facts(text: str) -> set[Fact]:
    facts = {
        Fact("date", match.group(0))
        for match in DATE_PATTERN.finditer(text)
    }
    for match in QUANTITY_PATTERN.finditer(text):
        facts.add(canonical_fact(
            match.group("value"),
            match.group("unit"),
        ))
    return facts

For claim C03, this converts 1 minute into time_seconds=60. The cited document contains 60-second, which becomes the same signature. The wording differs, but the values compare cleanly.

Decide and Explain Every Claim

Combine the citation gates and fact comparison in one function. The order matters: never read evidence through an unknown ID, and never treat a text-only claim as numerically supported.

def evaluate_claim(claim, documents):
    citation = claim["cited_doc_id"].strip()
    retrieved = split_ids(claim["retrieved_doc_ids"])

    if not citation:
        return "missing_citation", "claim has no citation"
    if citation not in documents:
        return "unknown_citation", f"{citation} is not in the handbook"
    if citation not in retrieved:
        return (
            "citation_not_retrieved",
            f"{citation} was not supplied in retrieved_doc_ids",
        )

    claim_facts = extract_facts(claim["claim_text"])
    evidence_facts = extract_facts(documents[citation])

    if not claim_facts:
        return (
            "needs_review_no_quantity",
            "no supported number-unit or YYYY-MM-DD pattern was found",
        )

    missing = claim_facts - evidence_facts
    if missing:
        labels = ", ".join(
            fact.label() for fact in sorted(missing)
        )
        return (
            "unsupported_quantity",
            f"not in cited evidence: {labels}",
        )

    labels = ", ".join(
        fact.label() for fact in sorted(claim_facts)
    )
    return "supported", f"matched cited evidence: {labels}"

Set difference makes the final rule compact. claim_facts - evidence_facts contains every normalized fact asserted by the claim but absent from the citation. The reason field preserves those missing signatures for inspection.

Run all cases and assert that the implementation reaches each expected path. Assertions make a later regex change fail loudly if it changes a labeled result during normal development runs:

results = []

for claim in claim_rows:
    status, reason = evaluate_claim(claim, documents)
    assert status == claim["expected_status"]
    results.append({
        **claim,
        "status": status,
        "reason": reason,
    })

In a production evaluator, the expected label belongs in a separate reviewed test set. New live claims will not have one. You run assertions during development, then apply the verified function to unlabeled saved outputs.

Read the Executed Results

The complete artifact builder executed all 16 cases and printed these counts:

claims=16 matched_expected=16
supported=7
unsupported_quantity=5
missing_citation=1
unknown_citation=1
citation_not_retrieved=1
needs_review_no_quantity=1

matched_expected=16 means the code reached the intended outcome for every synthetic test. It is a software-test result, not a claim that the method understands all language.

Five selected rows show why the reason matters:

C01: supported | matched cited evidence: time_seconds=15
C02: unsupported_quantity | not in cited evidence: time_seconds=45
C03: supported | matched cited evidence: time_seconds=60
C10: needs_review_no_quantity | no supported number-unit or YYYY-MM-DD pattern was found
C16: unsupported_quantity | not in cited evidence: date=2026-10-31

C01 is a direct match. C02 cites the same document, but 45 seconds is absent. C03 passes because 1 minute and 60 seconds share a normalized value. C10 contains no quantity, so this checker refuses to approve it. C16 cites a real retrieved document, but its date differs from the evidence.

Read the bars as work queues. The five red cases need their quantities removed or corrected. The three citation failures need a citation, retrieval, or document-ID repair. The blue text-only case needs another evaluation method or a reviewer. The seven green cases passed this numeric evidence rule only.

To use the checker in a real RAG pipeline, save one row per claim with the claim text, cited chunk ID, and exact retrieved chunk IDs. Keep document versions stable for each run. Record the prompt, model, retriever, and knowledge-base version so you can reproduce a failure after any component changes.

What Can Go Wrong

A matching number is treated as proof of the whole sentence. The claim “do not stop at 40 degrees Celsius” contains the same number as “stop at 40 degrees Celsius.” This checker can pass the quantity while missing the reversed meaning. Name the result numeric_check_passed in a production schema, or combine it with reviewed semantic evaluation.

Numbers are checked against all retrieved text. A value in an unrelated chunk can hide a bad citation. Compare each claim only with its cited evidence. If one claim cites several chunks, define whether one chunk or their combined text must support it.

Unit conversion changes the meaning. Time conversions are exact in this lesson. Temperature conversion, storage units, currencies, and rates may require context and careful rounding. Add one category at a time and test boundary cases before accepting it.

A range is reduced to two isolated numbers. Evidence that says “between 2 and 8 degrees Celsius” contains both endpoints. A claim of “5 degrees Celsius” may be valid under the range, but this exact-membership checker will flag it. Add explicit range logic instead of assuming every value between two extracted numbers is allowed.

Bare numbers disappear. The pattern intentionally ignores version 3.2 because it has no recognized unit. Add a version pattern if versions matter. Do not accept every bare number; document IDs and list numbers can create false matches.

Text-only claims pass by default. They should not. needs_review_no_quantity keeps the limitation visible and prevents a narrow tool from being reported as a general grounding checker.

The regex behaves differently after an edit. Keep labeled examples for hyphens, decimals, abbreviations, dates, unsupported units, and text-only claims. Run all assertions after every pattern change.

A date-shaped string is treated as a real date. The pattern rejects impossible month numbers, but it does not reject impossible calendar dates such as 2026-02-31. If calendar validity matters, parse each match with datetime.date.fromisoformat() and test leap years and month boundaries.

Assertions are disabled in an optimized Python run. The lesson uses assert for small development tests. In a production job, replace those checks with a test suite or explicit validation that cannot be skipped with python -O.

The chart fails on a server. Select Matplotlib’s non-interactive Agg backend before importing matplotlib.pyplot, then save the figure to a file. The official Matplotlib backends guide lists Agg as a non-interactive renderer suitable for file output.

Recap: Make the Narrow Check Explicit

Unsupported numbers can hide behind valid-looking citations. A transparent guardrail should first verify that a citation exists, names a known document, and belongs to the retrieved context. It can then extract quantities, normalize equivalent units, and show exactly which claimed signatures are absent from the cited text.

The executed lesson caught five altered quantities, three different citation problems, and one claim outside its numeric scope. It also accepted 1 minute against evidence that said 60 seconds because both values normalized to the same signature.

Keep the boundary honest: this method checks numeric evidence, not the full meaning or truth of an answer. Use its reason codes for fast regression tests and review queues, then add range logic, domain-specific units, and independent semantic review only when your real cases require them.