Lesson 3 - Prompt & Model Regression Tests in CI

Welcome to Prompt & Model Regression Tests in CI#

In Lessons 1 and 2 you learned to watch Docent in production and to catch drift and regressions after they happen. This lesson is about stopping the regression before it ever reaches production. Here is the situation it defends against: an engineer edits Docent’s system prompt to make answers a little friendlier, or upgrades the model to a newer version, runs the change past two questions by hand, sees good answers, and merges. What they didn’t check is the out-of-scope question — and the new prompt made Docent so eager to help that it now cheerfully invents a MongoDB aggregation feature Meridian doesn’t have. A third of the answers quietly got worse, and nobody found out until a user did.

The fix is to stop trusting prompts and models and start testing them. You already have the pieces from earlier modules: a golden set, metrics, and thresholds. This lesson wires them into your continuous integration (CI) pipeline as an automated test that runs on every change and fails the build — blocks the merge — when quality drops. That single move turns “we’ll eyeball it” into an enforced gate. A prompt change is now code, and code doesn’t ship until the tests are green.

In this lesson, you will:

• See why a prompt edit or model upgrade needs the same gate as any code change — an automated test that blocks merge on a quality drop
• Learn the anatomy of an LLM regression test: fixed golden cases, a metric per case, critical cases that must never fail, a per-suite threshold, and a clear pass/fail
• Build a pytest-style regression suite for Docent and watch it PASS on a good version, then BLOCK the release when a candidate regresses two critical cases
• Handle non-determinism the practical way — prefer deterministic metrics, pin the golden set, keep the suite fast — and run the deterministic suite twice for byte-identical results, plus one live-backed critical check

A prompt change is a code change#

When you change a function’s logic, you expect the test suite to run and to stop the merge if something breaks. A prompt is logic too — the difference is that its “bug” is a subtly worse answer instead of a stack trace, so it slips through code review that a crash never would. A model upgrade is even sneakier: the diff is one string (claude-haiku-4-5 → something else), yet it can shift tone, verbosity, and refusal behavior across your whole app at once. Neither of these is visible in a git diff, and neither is reliably caught by a human skimming a couple of example outputs.

So we borrow the discipline that already works for code: an automated gate. Every proposed change to Docent’s prompt or model triggers an eval suite; the suite scores a fixed set of cases and compares the result to a threshold; if quality is below the bar, the CI build goes red and the pull request cannot merge. The value isn’t just catching the regression — it’s that no one has to remember to look. The check runs on every change, and a reviewer who’s in a hurry can’t wave a regression through, because the build itself refuses.

Anatomy of an LLM regression test#

A regression test for an LLM app has five parts. Get these right and the rest is plumbing.

• Fixed golden cases. A pinned set of inputs with known-correct expectations — the same cases every run, changed only through deliberate review. If the set moves around, a “regression” might just be a different question, and the gate means nothing. This is the golden set you built back in the Building Eval Datasets module, now put to work.
• A metric per case. Each case needs a rule that turns an answer into a score. Prefer deterministic checks — an exact-match, a normalized substring, a whole-number match, a JSON-schema check — because they give the same verdict every time. A judge-based score is fine where you need it, but it’s a source of noise you’ll have to tame (more on that below).
• Critical cases. A subset that must never fail, regardless of the overall average. The out-of-scope refusal is a classic: it’s fine for Docent to be a little terse, but it is never fine for it to invent a feature. Every past incident should also become a critical case — a bug you fixed once should never be allowed to come back silently.
• A per-suite threshold. One rule that converts scores into a pass/fail for the whole suite. A good default has two clauses: an aggregate bar (overall >= 0.85) and a hard constraint (no critical case regresses). The average alone isn’t enough — a suite can average 0.9 while the one answer that matters most is wrong.
• A clear pass/fail. The suite exits green or red, and the red case names exactly what broke. CI acts on that exit status: green merges, red blocks.

Building the suite: golden cases and deterministic metrics#

Let’s build a real regression suite for Docent. We’ll start with the golden cases and their metrics. To keep the suite’s verdict reproducible, every metric here is deterministic: normalized substring checks and a whole-number match. Notice the two critical cases — the out-of-scope refusal, and a past bug (Docent once answered the Free-plan rate limit with the Pro number) turned into a permanent regression test.

import warnings
warnings.filterwarnings("ignore")
import re

def norm(s):
    return " ".join(s.lower().split())

def contains_all(*needles):
    def check(answer):
        a = norm(answer)
        return 1.0 if all(norm(n) in a for n in needles) else 0.0
    return check

def has_number(n, *also):
    # Whole-number match so "60" does NOT match inside "600" -- deterministic and precise.
    def check(answer):
        a = norm(answer)
        num_ok = re.search(rf"\b{n}\b", a) is not None
        words_ok = all(norm(w) in a for w in also)
        return 1.0 if (num_ok and words_ok) else 0.0
    return check

def is_refusal(answer):
    a = norm(answer)
    return 1.0 if "don't have that in the docs" in a else 0.0

# Each golden case: id, question, a metric (returns 1.0/0.0), and a critical flag.
GOLDEN = [
    {"id": "ratelimits-pro-rpm", "q": "How many requests per minute does the Pro plan allow?",
     "metric": has_number(600), "critical": False},
    {"id": "plans-pro-cost", "q": "How much does the Pro plan cost per month?",
     "metric": has_number(25), "critical": False},
    {"id": "errors-401", "q": "What does a 401 error mean?",
     "metric": contains_all("missing", "invalid"), "critical": False},
    {"id": "regions-fixed", "q": "Can you change a database's region after it is created?",
     "metric": contains_all("cannot"), "critical": False},
    # CRITICAL #1: the out-of-scope probe MUST be refused, never answered.
    {"id": "out-of-scope-refusal", "q": "Does Meridian support MongoDB-style aggregation pipelines?",
     "metric": is_refusal, "critical": True},
    # CRITICAL #2: a known past bug turned into a regression test -- Docent once
    # answered the Free-plan rate limit with the Pro number (600). It must say 60.
    {"id": "regression-free-plan-rpm", "q": "How many requests per minute does the Free plan allow?",
     "metric": has_number(60), "critical": True},
]

print(f"golden cases: {len(GOLDEN)}  (critical: {sum(c['critical'] for c in GOLDEN)})")

golden cases: 6  (critical: 2)

Six cases, two of them critical. Each metric returns a hard 1.0 or 0.0, so the score for a given answer never wavers between runs — which is exactly the property we need for a build gate.

The pytest-style runner and the threshold gate#

Now the runner. In a real repo you’d write these as functions named test_... and let pytest collect them; here we implement a tiny collector so the whole thing runs with plain python3 and stays self-contained, but the shape is the same — one test per golden case, plus two gate tests for the threshold, each producing a PASSED/FAILED line and a summary. The suite runs against an answer_of mapping: a fixed answer set standing in for what a particular prompt-and-model version produces. Stubbing the version’s answers is what makes the suite’s pass/fail reproducible; in CI you’d populate this mapping by running the real Docent once and caching its answers, or by using deterministic metrics on live output.

OVERALL_THRESHOLD = 0.85

def run_suite(answer_of):
    """answer_of: question -> answer string (a fixed/stubbed version under test).
    Returns a list of (test_name, passed, detail) mimicking pytest collection."""
    results = []
    scores = []
    critical_fail = False
    for case in GOLDEN:
        ans = answer_of[case["q"]]
        score = case["metric"](ans)
        scores.append(score)
        passed = score == 1.0
        tag = "critical" if case["critical"] else "case"
        detail = "" if passed else f"{'CRITICAL ' if case['critical'] else ''}regressed on '{case['id']}': got {ans!r}"
        if case["critical"] and not passed:
            critical_fail = True
        results.append((f"test_{tag}[{case['id']}]", passed, detail))
    overall = sum(scores) / len(scores)
    # Gate 1: overall score must clear the threshold.
    g1 = overall >= OVERALL_THRESHOLD
    results.append((f"test_gate[overall_score>={OVERALL_THRESHOLD}]", g1,
                    "" if g1 else f"overall {overall:.3f} < {OVERALL_THRESHOLD}"))
    # Gate 2: no critical case may regress.
    g2 = not critical_fail
    results.append(("test_gate[no_critical_regression]", g2,
                    "" if g2 else "one or more CRITICAL cases failed"))
    return results, overall

def report(results, overall, version_label):
    print("===================== test session starts =====================")
    print(f"suite: docent regression   version: {version_label}   collected {len(results)} items\n")
    for name, passed, detail in results:
        print(f"{name} {'PASSED' if passed else 'FAILED'}")
    n_fail = sum(1 for _, p, _ in results if not p)
    n_pass = len(results) - n_fail
    print()
    if n_fail:
        print("--------------------------- FAILURES ---------------------------")
        for name, passed, detail in results:
            if not passed:
                print(f"FAILED {name}\n    {detail}")
        print()
    print(f"overall score: {overall:.3f}   threshold: {OVERALL_THRESHOLD}")
    print(f"===================== {n_fail} failed, {n_pass} passed =====================")
    release_ok = n_fail == 0
    print("RELEASE: MERGE ALLOWED" if release_ok else "RELEASE: BLOCKED  (fix before merge)")
    return release_ok

The two test_gate[...] entries are the enforcement: one checks the aggregate, one checks the critical constraint, and CI treats any FAILED as a red build. Everything above is pure Python and pure lookups, so the same version under test yields the same report every time.

The suite PASSES on a good version#

Here is the current, trusted version of Docent as a fixed answer set — the answers a good prompt on claude-haiku-4-5 produces for our six golden questions. We run the suite twice to prove the deterministic verdict is byte-for-byte stable:

GOOD_ANSWERS = {
    "How many requests per minute does the Pro plan allow?": "The Pro plan allows 600 requests per minute.",
    "How much does the Pro plan cost per month?": "The Pro plan costs 25 dollars per month.",
    "What does a 401 error mean?": "A 401 error means a missing or invalid API key.",
    "Can you change a database's region after it is created?":
        "No. The region is fixed at creation and cannot be changed; you must export and re-import into a new database.",
    "Does Meridian support MongoDB-style aggregation pipelines?": "I don't have that in the docs.",
    "How many requests per minute does the Free plan allow?": "The Free plan allows 60 requests per minute.",
}

print("########## GOOD VERSION -- RUN 1 ##########")
res, ov = run_suite(GOOD_ANSWERS)
report(res, ov, "prompt@v3 (current)")

print("\n########## GOOD VERSION -- RUN 2 (must be identical) ##########")
res, ov = run_suite(GOOD_ANSWERS)
report(res, ov, "prompt@v3 (current)")

########## GOOD VERSION -- RUN 1 ##########
===================== test session starts =====================
suite: docent regression   version: prompt@v3 (current)   collected 8 items

test_case[ratelimits-pro-rpm] PASSED
test_case[plans-pro-cost] PASSED
test_case[errors-401] PASSED
test_case[regions-fixed] PASSED
test_critical[out-of-scope-refusal] PASSED
test_critical[regression-free-plan-rpm] PASSED
test_gate[overall_score>=0.85] PASSED
test_gate[no_critical_regression] PASSED

overall score: 1.000   threshold: 0.85
===================== 0 failed, 8 passed =====================
RELEASE: MERGE ALLOWED

########## GOOD VERSION -- RUN 2 (must be identical) ##########
===================== test session starts =====================
suite: docent regression   version: prompt@v3 (current)   collected 8 items

test_case[ratelimits-pro-rpm] PASSED
test_case[plans-pro-cost] PASSED
test_case[errors-401] PASSED
test_case[regions-fixed] PASSED
test_critical[out-of-scope-refusal] PASSED
test_critical[regression-free-plan-rpm] PASSED
test_gate[overall_score>=0.85] PASSED
test_gate[no_critical_regression] PASSED

overall score: 1.000   threshold: 0.85
===================== 0 failed, 8 passed =====================
RELEASE: MERGE ALLOWED

All eight items pass, the overall is a perfect 1.000, and both runs are identical down to the character. This is what a green CI build looks like: the change is safe to merge, and no human had to eyeball a thing.

Introduce a regression: the build BLOCKS#

Now the scenario from the intro. A well-meaning “v4 candidate” prompt made Docent more eager — and in doing so it hallucinates on the out-of-scope probe (critical #1) and resurfaces the old Free-plan bug, answering 600 instead of 60 (critical #2). Only those two answers change; everything else is still correct. Watch the suite catch it:

# The v4-candidate prompt accidentally made Docent answer everything, including
# the out-of-scope probe, and the Free-plan past-bug came back.
REGRESSED_ANSWERS = dict(GOOD_ANSWERS)
REGRESSED_ANSWERS["Does Meridian support MongoDB-style aggregation pipelines?"] = \
    "Yes, Meridian supports MongoDB-style aggregation pipelines via the /aggregate endpoint."
REGRESSED_ANSWERS["How many requests per minute does the Free plan allow?"] = \
    "The Free plan allows 600 requests per minute."

print("########## CANDIDATE VERSION (regressed) ##########")
res, ov = run_suite(REGRESSED_ANSWERS)
report(res, ov, "prompt@v4-candidate")

########## CANDIDATE VERSION (regressed) ##########
===================== test session starts =====================
suite: docent regression   version: prompt@v4-candidate   collected 8 items

test_case[ratelimits-pro-rpm] PASSED
test_case[plans-pro-cost] PASSED
test_case[errors-401] PASSED
test_case[regions-fixed] PASSED
test_critical[out-of-scope-refusal] FAILED
test_critical[regression-free-plan-rpm] FAILED
test_gate[overall_score>=0.85] FAILED
test_gate[no_critical_regression] FAILED

--------------------------- FAILURES ---------------------------
FAILED test_critical[out-of-scope-refusal]
    CRITICAL regressed on 'out-of-scope-refusal': got 'Yes, Meridian supports MongoDB-style aggregation pipelines via the /aggregate endpoint.'
FAILED test_critical[regression-free-plan-rpm]
    CRITICAL regressed on 'regression-free-plan-rpm': got 'The Free plan allows 600 requests per minute.'

overall score: 0.667   threshold: 0.85
===================== 4 failed, 4 passed =====================
RELEASE: BLOCKED  (fix before merge)

The build is red and the merge is blocked. Two things did the blocking, and both matter. The overall score fell to 0.667, tripping the aggregate gate — but even if only one critical case had regressed and the average had stayed above 0.85, the test_gate[no_critical_regression] clause would still have failed the build on its own. That’s the whole point of critical cases: the out-of-scope hallucination is not the kind of thing you let through because the average looked fine. The failure report names exactly what broke and shows the offending answers, so the engineer knows to fix the prompt (or roll it back) before trying to merge again. The regression never reaches a user.

Non-determinism, speed, and one live check#

The suite above is fully deterministic because the answers under test are pinned and the metrics are exact string checks. Real CI faces live model output, which varies run to run, so keep a few practices in mind:

• Prefer deterministic metrics. Exact-match, normalized substring, whole-number, and JSON-schema checks give a stable verdict. Lean on these for anything you want to gate on.
• Tame judge-based scores. When a case genuinely needs an LLM judge, don’t gate on a single noisy number: average several runs, gate on a threshold with a tolerance band (fail only on a clear drop, not a 0.01 wobble), and pin the judge’s model and prompt so the grader doesn’t drift underneath you.
• Pin the golden set. The cases are a versioned file. A regression should only ever come from a change to the prompt or model — never from the test set shifting.
• Keep it fast and cheap. CI runs on every push, so a suite of dozens of small cases with short max_tokens is the target, not thousands of long generations. Run the expensive full evaluation nightly; run the fast critical suite on every change.

Even so, a deterministic suite can’t tell you whether the real Docent still refuses the out-of-scope question today. For that you add a small number of live-backed checks — real calls whose pass/fail is still reproducible because the correct behavior is unambiguous. Here is one: call Docent live on the out-of-scope probe and assert it refuses.

import warnings
warnings.filterwarnings("ignore")
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

DOCS = [
    {"id": "ratelimits", "title": "Rate Limits",
     "text": "The Free plan allows 60 requests per minute. The Pro plan allows 600 requests per minute. Exceeding the "
             "limit returns HTTP 429 with a Retry-After header giving the seconds to wait. Rate limits are applied per "
             "API key, not per account."},
    {"id": "plans", "title": "Plans and Pricing",
     "text": "Meridian has three plans. Free costs 0 dollars per month and includes 1 GB of storage. Pro costs 25 dollars "
             "per month and includes 50 GB of storage. Scale costs 199 dollars per month and includes 500 GB of storage. "
             "All plans include unlimited read replicas."},
]

def retrieve(question, docs, k=2):
    q = set(question.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())), reverse=True)
    return scored[:k]

def docent_answer(question, docs, model="claude-haiku-4-5"):
    ctx = "\n\n".join(f"[{d['id']}] {d['title']}\n{d['text']}" for d in retrieve(question, docs))
    prompt = (f"Answer the question using ONLY the context. If the context does not contain the answer, say "
              f"\"I don't have that in the docs.\" Be concise.\n\nContext:\n{ctx}\n\nQuestion: {question}")
    msg = client.messages.create(model=model, max_tokens=200,
                                 messages=[{"role": "user", "content": prompt}])
    return msg.content[0].text

# One live-backed critical test: the out-of-scope probe must be refused.
q = "Does Meridian support MongoDB-style aggregation pipelines?"
answer = docent_answer(q, DOCS)
print("live answer:", repr(answer))

passed = "don't have that in the docs" in " ".join(answer.lower().split())
print(f"test_critical_live[out-of-scope-refusal] {'PASSED' if passed else 'FAILED'}")

live answer: "I don't have that in the docs."
test_critical_live[out-of-scope-refusal] PASSED

That is a genuine call to claude-haiku-4-5. The exact wording could vary slightly on a re-run — the model might phrase a refusal a hair differently — but the pass/fail is stable because the metric only asks the one thing that must hold: does Docent refuse rather than invent an answer? Deterministic metrics on live output are how you get a reproducible gate without pinning the model’s every word. (The key, as always, is read from the environment by the client and never appears in the code, the output, or a log.)

Practice Exercises#

Exercise 1: Add a new critical case from an incident#

Suppose production monitoring caught Docent claiming the Free plan includes backups retained for 30 days (the real Free retention is 7 days; 30 is the Pro number). Turn that incident into a permanent critical regression test: add a golden case for the question “How long are backups retained on the Free plan?” with a deterministic metric that requires the answer to contain 7 (as a whole number) and not contain 30, marked critical: True. Re-run the good and regressed answer sets and confirm the new case behaves.

Exercise 2: Make the aggregate gate stricter than the critical gate#

Change OVERALL_THRESHOLD to 0.90 and construct a version whose answers pass every critical case but let two non-critical cases regress (e.g. drop the regions-fixed and errors-401 answers to wrong text). Run the suite and confirm the build blocks on the aggregate gate alone, even though no critical case failed. This shows the two gate clauses catch different failure modes.

Exercise 3: Simulate judge noise and add a tolerance band#

Replace one case’s deterministic metric with a stubbed “judge” that returns a score drawn from a small fixed list of values (e.g. [0.86, 0.88, 0.90, 0.84, 0.87]) to imitate run-to-run variation. Average N draws and gate that case on mean >= 0.85 - 0.02 (a tolerance band) so a tiny wobble doesn’t flip the build. Discuss why gating on a single judge call would make this suite flaky.

Summary#

You turned Docent’s eval suite into a CI gate that treats prompts and model choices like code. A prompt edit or a model upgrade is invisible in a git diff and unreliable to eyeball, so you built an automated test that runs on every change: six fixed golden cases, a deterministic metric per case, two critical cases that must never fail (the out-of-scope refusal and a past bug turned into a regression test), and a two-clause threshold — overall >= 0.85 and no critical case regresses. On the good version the suite passed all eight items with a perfect 1.000 and produced byte-identical reports across two runs; on a regressed candidate it caught the hallucinated out-of-scope answer and the resurfaced Free-plan bug, dropped the overall to 0.667, failed both gate tests, and printed RELEASE: BLOCKED. You also saw how to handle non-determinism — prefer deterministic metrics, tame judge scores with averaging and tolerance, pin the golden set, keep it fast — and ran one live-backed critical check that Docent still refuses the out-of-scope probe, all without the key ever appearing anywhere.

Key Concepts#

• A prompt/model change is a code change — gate it behind an automated test that fails the build on a quality drop, so no one has to remember to check and no reviewer can wave a regression through.
• The five parts of an LLM regression test — fixed golden cases, a metric per case, critical cases, a per-suite threshold, and a clear pass/fail that CI acts on.
• Critical cases and a two-clause threshold — an aggregate bar catches broad drops; a “no critical case regresses” clause catches the single unacceptable failure an average would hide. Every past incident becomes a critical case.
• Determinism is what makes a gate trustworthy — deterministic metrics on pinned inputs give byte-identical verdicts; for judge-based scores, average multiple runs, add a tolerance band, and pin the judge model and prompt.
• On red, fix or roll back — don’t weaken the gate — lowering the threshold to ship a known regression defeats the mechanism; only change the golden set through deliberate review.

Why This Matters#

Almost every silent quality regression in a shipped LLM app traces back to a change no one thought was risky — a friendlier prompt, a newer model, a reworded instruction — merged after a quick manual look. A regression suite in CI is what converts that hope into a guarantee: the change either clears a known bar and the critical behaviors hold, or it doesn’t merge. That’s the difference between finding out your assistant started hallucinating features from a customer complaint versus from a red build on the pull request that introduced it. It also compounds over time — every incident you promote to a critical case makes the gate a little stronger, so the same bug can never come back twice. Teams that test their prompts ship faster and safer, because they can change with confidence instead of changing and praying.

Continue Building Your Skills#

You now have a gate that stops a bad change from ever merging — the eval suite runs in CI, the build goes red when quality drops or a critical case regresses, and the regression never reaches production. But a green build is a promise about the cases you tested, not a shield against everything a real user might send or the model might occasionally produce at runtime. In Lesson 4 you’ll add guardrails — input and output checks that block off-topic prompts, unsafe requests, and malformed answers as they happen — and close the loop with human feedback that flows the thumbs-down and the correction back into the golden set. The critical cases you wrote today are exactly where that feedback lands: every real failure a user catches becomes tomorrow’s regression test, and the gate gets stronger with every incident.