Lesson 2 - Detecting Drift & Regressions

Welcome to Detecting Drift & Regressions#

In Lesson 1 you stood up continuous monitoring for Docent — our Meridian documentation assistant — and watched the dashboards: request rate, latency, cost, refusal rate, a rolling quality proxy. Dashboards tell you what the numbers are right now. This lesson is about the harder skill: noticing when those numbers have moved in a way that means Docent is quietly getting worse — before a user files the ticket that tells you so.

The failure mode here is not a crash. It’s decay. Real Meridian users start asking about billing, a topic your golden set barely covers, and Docent’s answers on that slice quietly slide. Someone ships a “small” prompt tweak on a Friday and the recent window’s judge scores drop a fifth without a single error being thrown. The p95 latency graph stays flat, every request returns HTTP 200, and Docent is measurably worse than it was on Tuesday. Catching that is a matter of comparing windows — recent versus baseline — and deciding, with a threshold and a little statistics, whether the difference is a real shift or just sample noise. That’s what you’ll build here, deterministically, so it flags the same drift on every run.

In this lesson, you will:

• Distinguish the three kinds of change: input/data drift (the questions moved), quality/behavior drift (the answers got worse), and prompt/version drift (the thing under you changed) — and separate a gradual drift from a sudden regression
• Learn the core detection moves: compare metric windows (recent vs baseline), monitor the input distribution for new topics, canary/online-sample live outputs, and watch a rolling quality proxy or a rising refusal/thumbs-down rate
• Tell signal from noise — small-sample wobble versus a real shift — with a threshold and a simple significance check
• Build a deterministic detector that flags an input drift (a billing-question surge) and a quality regression (a windowed score drop) with real, reproducible numbers

Three kinds of change, and drift vs regression#

“Docent got worse” is too vague to act on. To fix decay you first have to name what changed, because the diagnosis and the fix are different for each. There are three things that can move, and one important distinction in how they move.

• Input / data drift — the questions changed. Your users start asking about things your test set never anticipated. Docent itself is unchanged; the world it faces isn’t. If a wave of billing questions arrives and your golden set is 90% auth-and-ratelimits, your evaluation is now measuring a slice of traffic that’s shrinking, and the growing slice is unmeasured. Nothing is broken yet, but you’ve gone partially blind.
• Quality / behavior drift — the answers changed. The same question that got a good answer last month gets a worse one now. This is what you ultimately care about, and it has two common causes: something underneath shifted (see the next point), or input drift pushed traffic toward a slice Docent was always weak on and you only now feel it.
• Prompt / version drift — the system changed. Someone edited the prompt template, swapped the model version, or changed the retrieval logic. Each of these can move behavior without touching the request-handling code, which is exactly why it’s so easy to ship one by accident and so important to capture the prompt and model per call (Module 7) so you can attribute a behavior change to the change that caused it.

Cutting across all three is the distinction between drift and a regression:

• A regression is a step change tied to a specific event — a deploy, a prompt edit, a model bump. The quality graph has a visible elbow: fine on one side, worse on the other, with a timestamp you can line up against your deploy log. Regressions are the good kind of bad news, because they’re attributable: find the change, and you can roll it back.
• Drift is gradual, with no single cause — user topics slowly shift over weeks, or a retrieval index slowly staleness-rots. There’s no elbow to point at, just a slope. Drift is harder because there’s nothing to roll back; you have to adapt (expand the golden set, refresh the index, re-tune the prompt).

How to detect it: compare windows, watch the distribution, sample live#

You can’t eyeball a drift into existence; you detect it by comparison. Every method below reduces to “is the recent period different enough from a trusted baseline to act on?”

• Compare metric windows (recent vs baseline). Pick a baseline window you trust — say, the week after your last good evaluation — and a recent window — the last day or last N requests. Compute the same metric over both (mean judge score, refusal rate, pass rate) and look at the difference. A window smooths out per-request noise, and the baseline gives you something to be different from. This is the workhorse; the other methods are variations on it.
• Monitor the input distribution. Bucket incoming questions into topics and track the proportion in each bucket over time. A brand-new bucket appearing (billing), or an existing one ballooning, is input drift you can see before it shows up as a quality drop — an early warning that your golden set is falling out of sync with reality.
• Canary / online sampling. You can’t judge every live answer, but you can judge a random sample of them continuously — a canary. Run your Module 3 LLM judge (or a cheap deterministic proxy) over, say, 2% of live traffic, and you get a rolling quality estimate on real questions, not just your frozen golden set. This is how you notice quality drift on the slices your test set never covered.
• Watch a rolling quality proxy or a refusal/feedback signal. Even without a judge, cheap proxies move when quality moves: a climbing refusal rate (“I don’t have that in the docs”) can signal that retrieval has drifted and is no longer surfacing the right pages; a rising thumbs-down rate is users telling you directly. These are lagging and noisy, but they’re free and they catch things.

The unifying idea is that a single number is meaningless without a baseline to compare it to. “The mean score is 0.66” tells you nothing; “the mean score is 0.66, down from a baseline of 0.87” tells you to page someone.

Detecting input drift: a distribution shift#

Let’s make the left panel real. We have topic category counts for two windows of Docent traffic — a trusted baseline and the recent window. We turn each into proportions (so the two windows are comparable even at different volumes), then measure how far apart the two distributions are with total-variation distance — half the sum of absolute differences in each category’s share, a clean number in $[0, 1]$ where 0 is identical and 1 is completely disjoint. We also explicitly check for new categories that didn’t exist in the baseline at all, because a brand-new topic is the sharpest drift signal there is.

import warnings
warnings.filterwarnings("ignore")

# Category counts of the topics users asked about, in two windows.
baseline_topics = {"auth": 40, "ratelimits": 25, "plans": 20, "errors": 10, "backups": 5}
recent_topics   = {"auth": 22, "ratelimits": 18, "plans": 14, "errors": 8,  "backups": 4, "billing": 34}

def to_proportions(counts):
    total = sum(counts.values())
    return {k: v / total for k, v in counts.items()}

def total_variation(p, q):
    cats = set(p) | set(q)
    return 0.5 * sum(abs(p.get(c, 0.0) - q.get(c, 0.0)) for c in cats)

def new_categories(baseline, recent):
    return sorted(set(recent) - set(baseline))

p_base, p_recent = to_proportions(baseline_topics), to_proportions(recent_topics)
tvd = total_variation(p_base, p_recent)
new_cats = new_categories(baseline_topics, recent_topics)
new_share = sum(recent_topics[c] for c in new_cats) / sum(recent_topics.values())

TVD_THRESHOLD = 0.15
input_drift = tvd > TVD_THRESHOLD or len(new_cats) > 0

print(f"total variation distance : {round(tvd, 4)}  (threshold {TVD_THRESHOLD})")
print(f"new categories in recent : {new_cats}")
print(f"share of recent traffic in new topics : {round(new_share, 4)}")
print(f"INPUT DRIFT FLAGGED : {input_drift}")

total variation distance : 0.34  (threshold 0.15)
new categories in recent : ['billing']
share of recent traffic in new topics : 0.34
INPUT DRIFT FLAGGED : True

The number to sit with is 0.34. More than a third of the recent window’s questions are about billing — a topic that did not exist in the baseline and that Docent’s golden set therefore does not cover at all. Docent might be answering those questions badly and your offline evaluation would never know, because your evaluation still runs on the old auth-heavy distribution. That’s the danger of input drift: it doesn’t make your metrics look bad, it makes them irrelevant, because they’re measuring a shrinking slice of what’s actually happening. The fix isn’t to roll anything back — it’s to notice billing arrived and get it into your golden set (which loops right back to Module 5’s “production logs become tomorrow’s test cases”). Because this is pure arithmetic over fixed counts, the number is exactly 0.34 every run — no model call, nothing to vary.

Detecting a quality regression: a windowed score drop#

Now the right panel. Suppose a prompt change shipped between two windows, and we have per-item judge scores in $[0, 1]$ from a canary that sampled live answers on each side of it (Module 3’s judge produced these; here we hold them fixed so the detector is reproducible). We compare the two windows’ means, flag a regression when the drop exceeds a threshold, and then — because a mean drop on a small sample could be luck — confirm it with a two-proportion z-test on the pass rate (treating a score of at least 0.8 as a “pass”). A large $|z|$ says the drop is bigger than sampling noise would plausibly produce.

# Per-item judge scores in [0, 1] for two windows: before and after a prompt change.
baseline_scores = [0.9, 0.8, 1.0, 0.7, 0.9, 0.8, 1.0, 0.9, 0.8, 0.9,
                   1.0, 0.7, 0.9, 0.8, 0.9, 1.0, 0.8, 0.9, 0.7, 0.9]
recent_scores   = [0.7, 0.6, 0.8, 0.5, 0.7, 0.6, 0.9, 0.7, 0.6, 0.5,
                   0.8, 0.6, 0.7, 0.5, 0.6, 0.8, 0.7, 0.6, 0.5, 0.7]

def mean(xs):
    return sum(xs) / len(xs)

def pass_rate(xs, bar=0.8):
    passes = sum(1 for x in xs if x >= bar)
    return passes, len(xs), passes / len(xs)

def two_proportion_z(x1, n1, x2, n2):
    p1, p2 = x1 / n1, x2 / n2
    p_pool = (x1 + x2) / (n1 + n2)
    se = (p_pool * (1 - p_pool) * (1 / n1 + 1 / n2)) ** 0.5
    return 0.0 if se == 0 else (p1 - p2) / se

base_mean, recent_mean = mean(baseline_scores), mean(recent_scores)
drop = base_mean - recent_mean
DROP_THRESHOLD = 0.05
regression = drop > DROP_THRESHOLD

b_pass, b_n, b_rate = pass_rate(baseline_scores)
r_pass, r_n, r_rate = pass_rate(recent_scores)
z = two_proportion_z(b_pass, b_n, r_pass, r_n)

print(f"baseline window : n={b_n} mean={round(base_mean, 4)} pass_rate={round(b_rate, 4)}")
print(f"recent window   : n={r_n} mean={round(recent_mean, 4)} pass_rate={round(r_rate, 4)}")
print(f"mean drop        : {round(drop, 4)}  (threshold {DROP_THRESHOLD})")
print(f"REGRESSION FLAGGED : {regression}")
print(f"two-proportion z : {round(z, 4)}  (|z| > 1.96 ~ significant at 95%)")
print(f"SIGNIFICANT DROP : {abs(z) > 1.96 and r_rate < b_rate}")

# A small window that only wobbled: same threshold must NOT fire.
noisy_baseline = [0.9, 0.8, 1.0, 0.7, 0.9]
noisy_recent   = [0.8, 0.9, 0.7, 0.9, 0.8]
n_drop = mean(noisy_baseline) - mean(noisy_recent)
print(f"small-window drop : {round(n_drop, 4)}  -> regression flagged: {n_drop > DROP_THRESHOLD}")

baseline window : n=20 mean=0.865 pass_rate=0.85
recent window   : n=20 mean=0.655 pass_rate=0.2
mean drop        : 0.21  (threshold 0.05)
REGRESSION FLAGGED : True
two-proportion z : 4.1161  (|z| > 1.96 ~ significant at 95%)
SIGNIFICANT DROP : True
small-window drop : 0.04  -> regression flagged: False

The mean fell from 0.865 to 0.655 — a drop of 0.21, four times the threshold — and the pass rate collapsed from 85% to 20%. The two-proportion z of 4.12 is far past 1.96, so this is not sampling luck; something real happened, and since we know a prompt change shipped between the windows, it’s a regression we can go roll back. The last two lines are the whole point of the significance check: a small five-item window that merely jiggled produced a drop of 0.04, under the threshold, and correctly did not fire. That’s the discipline — a threshold to ignore tiny moves, and a significance check to make sure a flagged move is bigger than noise. Every number here is deterministic arithmetic over fixed lists, so it prints identically on every run.

Tuning the alarm: sensitivity vs noise#

Both detectors hinge on choices — the TVD threshold, the score-drop threshold, the pass bar, the window size — and those choices trade sensitivity against false alarms. Set the drop threshold too low (0.005) and normal day-to-day wobble trips it constantly, your team learns to ignore the alerts, and the one real regression gets ignored with them. Set it too high (0.30) and Docent can degrade badly before anything fires. The same tension governs window size: a short recent window reacts fast but is noisy (few samples), while a long window is stable but sluggish to notice a sudden regression.

There’s no universal right answer, but there is a right method: set thresholds from your baseline’s own variability. If your baseline mean score bounces around by ±0.03 week to week when nothing has changed, a 0.03 threshold is pure noise — pick something comfortably above that natural wobble (0.05–0.08). And always pair the threshold with the significance check from the last section, so that even when a threshold is crossed, a tiny-sample fluke doesn’t page anyone. The goal is an alarm that stays quiet through normal variation and fires reliably on a real shift — which is exactly the behavior the small-window drop : 0.04 -> False line demonstrated.

Practice Exercises#

Exercise 1: Would this alert, and should it?#

You’re monitoring Docent’s mean judge score. The baseline window (n=200) has mean 0.84. Three separate recent windows come in: (a) n=200, mean 0.80; (b) n=6, mean 0.63; (c) n=200, mean 0.835. Using a drop threshold of 0.05, say for each whether the threshold fires, then reason about whether you’d actually page someone once you also consider sample size. Which of the three is the clearest true regression, and which is the most likely false alarm?

Exercise 2: Add a refusal-rate drift check#

Docent refuses with “I don’t have that in the docs.” A climbing refusal rate can signal retrieval drift (the right pages stopped being surfaced). Given baseline_refusals = 8 out of baseline_total = 200 and recent_refusals = 44 out of recent_total = 200, compute both refusal rates and reuse the two_proportion_z function from this lesson to test whether the rise is significant. Would you flag retrieval drift?

Exercise 3: Drift or regression?#

For each scenario, decide whether it’s input drift, quality/behavior drift, or a regression, and name the one signal from this lesson you’d look at to confirm it: (a) Docent’s mean score was flat for a month, then dropped sharply the same afternoon a new prompt was deployed. (b) Over eight weeks, the share of questions about plans and billing slowly rose from 10% to 45% while auth questions faded. (c) No deploy happened, but the mean score has slid gradually over three weeks and the refusal rate is creeping up. Explain your reasoning for the trickiest one.

Summary#

Quality in production rarely fails loudly — it decays, and catching decay means comparing a recent window against a trusted baseline and deciding whether the difference is real. Name the change first: input/data drift (the questions moved and your golden set no longer covers them), quality/behavior drift (the answers got worse), or prompt/version drift (the system changed underneath you); and distinguish a regression — a step change you can align to a deploy and roll back — from drift — a gradual slope with no single cause that you have to adapt to. Detect it by comparing metric windows, monitoring the input distribution for new topics, canary-sampling live outputs with a judge, and watching cheap proxies like refusal and thumbs-down rates. You built both halves deterministically: an input-drift detector that flagged a billing surge with a total-variation distance of 0.34 (a brand-new topic taking 34% of recent traffic), and a quality-regression detector that flagged a windowed mean score dropping from 0.865 to 0.655 — a 0.21 fall past the 0.05 threshold, confirmed by a two-proportion z of 4.12 — while correctly not firing on a five-item window that only wobbled by 0.04.

Key Concepts#

• Three kinds of change — input/data drift (the questions moved), quality/behavior drift (the answers got worse), and prompt/version drift (the system changed); each has a different fix.
• Drift vs regression — a regression is a step change tied to a specific deploy (attributable, reversible); drift is a gradual slope with no single cause (you adapt, not roll back). Line the quality graph up against the deploy log to tell them apart.
• Compare windows — a single metric value is meaningless without a baseline; every detection method reduces to “is the recent window different enough from a trusted baseline to act on?”
• Detection methods — compare metric windows, monitor the input distribution for new/growing topics, canary-sample live outputs with a judge, and watch a rolling quality proxy or a rising refusal/thumbs-down rate.
• Threshold and significance are both required — a threshold ignores moves too small to matter; a significance check (a two-proportion z, $|z| > 1.96$) ignores moves too small on too little data to trust. Requiring both keeps the alarm from crying wolf while still catching real shifts.
• Total-variation distance — half the sum of absolute per-category share differences, a clean drift score in $[0, 1]$; a brand-new category is the sharpest input-drift signal.

Why This Matters#

The regressions that hurt most in production are invisible to the monitors you built for ordinary services. A Friday prompt tweak that drops a fifth of Docent’s answer quality throws no exception, returns HTTP 200, and keeps p95 latency flat — the only evidence is that the recent window’s scores are lower than last week’s, and you only see it if you’re comparing windows and know how to tell a real shift from noise. Input drift is subtler still: it doesn’t make your metrics look bad, it makes them irrelevant, quietly measuring a shrinking slice of traffic while a new billing topic Docent was never evaluated on takes over. The discipline in this lesson — baseline versus recent, a threshold to skip the trivial, a significance check to skip the noise — is what turns “users say Docent feels worse lately” into “the recent window’s pass rate fell from 85% to 20%, z=4.12, aligned to yesterday’s deploy: roll it back.” That is the difference between finding a regression yourself in an hour and having a customer find it for you in a week.

Continue Building Your Skills#

You can now tell the three ways Docent can silently get worse apart — the questions drifting away from your test set, the answers degrading, or the system changing under you — and separate a gradual drift from a sudden, attributable regression. More importantly, you built the detectors: a distribution-shift check that caught a billing surge your golden set never covered, and a windowed score comparison that caught a prompt-change regression and confirmed it was signal, not noise, with a two-proportion z. The through-line is comparison against a trusted baseline, guarded by a threshold and a significance check so the alarm is both sensitive and quiet. Lesson 3 takes this comparison out of the dashboard and into your pipeline: you’ll run your golden-set evaluation as a regression test in CI, so a prompt or model change that would drop quality fails the build and never reaches the users you’ve been monitoring here.