Lesson 3 - From Production Logs to Test Sets

Welcome to From Production Logs to Test Sets#

In Lesson 2 you wrote test cases for Docent, our documentation assistant for the Meridian serverless database, by hand — one careful question at a time, each with a reference answer and tags. Hand-writing is the right way to start, and it will always have a place for the cases you know matter. But it has a quiet failure mode: you write the questions you imagine users asking, not the questions they actually ask.

Real developers do not phrase things the way you do. They ask “what status code when i exceed the rate limit?” — lowercase, no punctuation, a double question mark — where you carefully wrote “What HTTP status code does Meridian return when you exceed the rate limit?” They misspell. They combine two questions into one. They ask about features Meridian does not have. A hand-written set that never sees this drifts away from reality, and your evaluation slowly starts measuring a version of Docent that only exists in your head.

Production logs are the fix. Every time Docent answers a real question, it produces a small record of exactly what a real user wanted and how they said it. This lesson is about turning that stream of records into a trustworthy test set: capturing the right fields, cutting the near-duplicates, sampling the failures on purpose, doing the hard work of labeling, and — the highest-value move of all — promoting past failures into regression cases so a bug you already caught can never sneak back in.

In this lesson, you will:

• Explain why hand-written test sets drift from how users really talk, and what production logs reveal that imagination misses
• Walk the five-stage pipeline: capture, deduplicate, sample, label, promote
• Run a real, deterministic pipeline that takes 15 logged Meridian turns down to 13 unique down to 8 labeled candidates
• Promote a past Docent failure into a frozen regression case with a correct reference answer
• Scrub secrets and PII before a log line ever becomes a fixture in your repo

Why hand-written sets drift#

A hand-written test set is a snapshot of your assumptions about Docent’s users on the day you wrote it. That snapshot ages. Meridian ships a new feature and users start asking about it; a confusing doc page generates a wave of near-identical questions; a whole class of users turns out to type in lowercase with no punctuation. None of that is in your set, because you could not have imagined the exact shape of it.

Production logs close the gap because they are not a guess. They are the record of what happened. Three things fall out of them that hand-writing almost never captures:

• Real phrasing. The same intent arrives a dozen ways — “How much is Pro?”, “pro plan cost per month?”, “How much does the Pro plan cost per month?”. If Docent handles one wording but fumbles another, only the logs will tell you.
• Real failures. When a user gives a thumbs-down, they have handed you a labeled bug for free: this question, this answer, marked wrong by the person who asked. That is the single most valuable row in the log.
• Real distribution. Logs show you which questions are common and which are rare, so you can weight your test set toward what actually matters instead of treating every topic as equally likely.

The catch is that raw logs are not a test set. They are noisy, repetitive, full of PII, and — crucially — unlabeled. You cannot score an answer against “the reference” when there is no reference. The pipeline below is how you turn the raw stream into something you can actually evaluate against.

The pipeline: capture, dedupe, sample, label, promote#

Turning logs into fixtures is a funnel. Many raw turns go in one end; a smaller set of clean, labeled, high-value test cases comes out the other. Five stages do the work.

Capture means logging every Docent turn as a structured record, not just the text. The fields that make a log useful later are the question, the answer, the retrieved doc ids, the user feedback (a thumbs up/down or a rating), and the operational numbers — latency and cost. You saw latency and tokens in Module 1; here they earn their keep, because a slow or expensive answer is a candidate worth testing even when nobody complained.

Deduplicate collapses near-identical questions. Ask a real audience one thing and you get the same question fifty times in forty wordings. If you keep them all, your test set over-weights whatever happened to be popular and wastes labeling effort on copies.

Sample picks which unique questions to promote. You never label all of them. A good sample mixes three strategies: random (a fair cross-section), stratified (deliberately spanning topics or slices so no area is missed), and targeted (every thumbs-down and low-rated turn, because those are your known failures).

Label is where a raw question becomes a test case: someone writes the correct reference answer and records which doc it comes from. This is almost always the hardest and slowest step, because it needs a human who knows the right answer — the log gives you the question for free but never the reference.

Promote moves the labeled cases into the golden set, and gives special treatment to the failures: a past thumbs-down becomes a regression case, frozen so that a future change to Docent cannot silently reintroduce the same bug.

Build the pipeline, for real#

Here is the whole funnel as runnable code. It starts from a small, clearly-synthetic slice of Docent’s production log — fifteen Meridian turns, with a few near-duplicate phrasings, three thumbs-down failures, and a couple of out-of-scope questions mixed in, exactly like real traffic. The code counts what came in, deduplicates near-identical questions with plain normalization, samples the failures plus a stratified slice of the rest, and emits a candidate test set. No API key and no model are involved, so it prints the same numbers every run.

import warnings, re
warnings.filterwarnings("ignore")

# A small, clearly-synthetic slice of Docent's production log. Each entry is one
# real-shaped turn: the user's question, Docent's answer, the doc ids retrieved,
# a thumbs feedback signal, latency, and cost. Near-duplicates, thumbs-down, and
# out-of-scope questions are all present on purpose.
PROD_LOG = [
    {"id": 1,  "question": "What status code do I get when I hit the rate limit?",
     "answer": "HTTP 429.", "retrieved": ["ratelimits", "errors"],
     "feedback": "up", "latency_s": 1.9, "cost_usd": 0.0007},
    {"id": 2,  "question": "what status code when i exceed the rate limit?",
     "answer": "HTTP 429.", "retrieved": ["ratelimits"],
     "feedback": None, "latency_s": 2.1, "cost_usd": 0.0007},
    {"id": 3,  "question": "What HTTP status code does Meridian return when you exceed the rate limit?",
     "answer": "429.", "retrieved": ["ratelimits"],
     "feedback": "up", "latency_s": 1.7, "cost_usd": 0.0006},
    {"id": 4,  "question": "How many requests per minute on the Pro plan?",
     "answer": "600 requests per minute.", "retrieved": ["ratelimits", "plans"],
     "feedback": "up", "latency_s": 2.0, "cost_usd": 0.0008},
    {"id": 5,  "question": "how many requests per minute does pro allow??",
     "answer": "600 per minute.", "retrieved": ["ratelimits"],
     "feedback": None, "latency_s": 1.8, "cost_usd": 0.0007},
    {"id": 6,  "question": "How much is the Pro plan per month?",
     "answer": "The Pro plan costs 25 dollars per month.", "retrieved": ["plans"],
     "feedback": "up", "latency_s": 1.6, "cost_usd": 0.0006},
    {"id": 7,  "question": "Can I change a database region after creating it?",
     "answer": "You can switch it anytime from the dashboard settings.", "retrieved": ["regions"],
     "feedback": "down", "latency_s": 2.3, "cost_usd": 0.0009},
    {"id": 8,  "question": "How long are backups kept on the Scale plan?",
     "answer": "90 days.", "retrieved": ["backups"],
     "feedback": "up", "latency_s": 1.9, "cost_usd": 0.0007},
    {"id": 9,  "question": "which env var does the python sdk read for the key?",
     "answer": "MERIDIAN_API_KEY.", "retrieved": ["sdks"],
     "feedback": "up", "latency_s": 1.5, "cost_usd": 0.0006},
    {"id": 10, "question": "what does a 401 error mean",
     "answer": "A missing or invalid API key.", "retrieved": ["errors"],
     "feedback": None, "latency_s": 1.7, "cost_usd": 0.0006},
    {"id": 11, "question": "Does Meridian support MongoDB-style aggregation pipelines?",
     "answer": "Yes, use the pipeline() helper to chain stages.", "retrieved": ["sdks"],
     "feedback": "down", "latency_s": 2.5, "cost_usd": 0.0011},
    {"id": 12, "question": "does meridian have graphql subscriptions?",
     "answer": "Yes, subscribe over the /graphql endpoint.", "retrieved": ["sdks"],
     "feedback": "down", "latency_s": 2.4, "cost_usd": 0.0010},
    {"id": 13, "question": "How long is the grace period before a deleted database is purged?",
     "answer": "24 hours.", "retrieved": ["deletion"],
     "feedback": "up", "latency_s": 1.8, "cost_usd": 0.0007},
    {"id": 14, "question": "How much does the pro plan cost per month?",
     "answer": "25 dollars per month.", "retrieved": ["plans"],
     "feedback": None, "latency_s": 1.6, "cost_usd": 0.0006},
    {"id": 15, "question": "Can you change a database's region after it is created?",
     "answer": "No, the region is fixed at creation; export and re-import to move.", "retrieved": ["regions"],
     "feedback": "up", "latency_s": 2.0, "cost_usd": 0.0008},
]

def normalize(text):
    """Lowercase, strip punctuation, collapse whitespace -> a bag of tokens."""
    text = text.lower()
    text = re.sub(r"[^a-z0-9\s]", " ", text)
    return set(text.split())

def near_duplicate(a, b, threshold=0.6):
    """Jaccard similarity of token sets; True when they overlap past the threshold."""
    ta, tb = normalize(a), normalize(b)
    if not ta or not tb:
        return False
    jaccard = len(ta & tb) / len(ta | tb)
    return jaccard >= threshold

def deduplicate(entries, threshold=0.6):
    """Keep the first log entry of each near-duplicate cluster."""
    unique = []
    for e in entries:
        if not any(near_duplicate(e["question"], u["question"], threshold) for u in unique):
            unique.append(e)
    return unique

# --- Step 1: what came in ---
total = len(PROD_LOG)

# --- Step 2: deduplicate near-identical questions ---
unique = deduplicate(PROD_LOG)

# --- Step 3: sample for the candidate test set ---
# Targeted: every thumbs-down (a real user told us it was wrong) is a must-keep.
# Stratified: also pull one representative from the remaining unique questions
# so the set is not only failures.
thumbs_down = [e for e in unique if e["feedback"] == "down"]
down_ids = {e["id"] for e in thumbs_down}
others = [e for e in unique if e["id"] not in down_ids]
sampled_others = others[::2]  # deterministic 1-in-2 stratified sample
candidates = thumbs_down + sampled_others

print(f"{total} logged -> {len(unique)} unique -> {len(candidates)} promoted candidates")
print()
print(f"thumbs-down kept (targeted): {sorted(down_ids)}")
print(f"sampled from the rest (stratified): {[e['id'] for e in sampled_others]}")
print()
print("Candidate test set (each still needs a reference answer to be labeled):")
for e in candidates:
    tag = "FAILURE" if e["feedback"] == "down" else "sample"
    print(f"  id={e['id']:>2}  [{tag:7}]  {e['question']}")

Running it prints the funnel and the candidate set:

15 logged -> 13 unique -> 8 promoted candidates

thumbs-down kept (targeted): [7, 11, 12]
sampled from the rest (stratified): [1, 4, 6, 9, 13]

Candidate test set (each still needs a reference answer to be labeled):
  id= 7  [FAILURE]  Can I change a database region after creating it?
  id=11  [FAILURE]  Does Meridian support MongoDB-style aggregation pipelines?
  id=12  [FAILURE]  does meridian have graphql subscriptions?
  id= 1  [sample ]  What status code do I get when I hit the rate limit?
  id= 4  [sample ]  How many requests per minute on the Pro plan?
  id= 6  [sample ]  How much is the Pro plan per month?
  id= 9  [sample ]  which env var does the python sdk read for the key?
  id=13  [sample ]  How long is the grace period before a deleted database is purged?

There is the whole funnel in three numbers: 15 logged → 13 unique → 8 promoted candidates. Because none of this touches the model, it is fully deterministic — run it a second time and every number and id is identical:

15 logged -> 13 unique -> 8 promoted candidates

Read the sample carefully and you learn something honest about the dedupe step. The three thumbs-down failures (ids 7, 11, 12) were all kept because we targeted them on purpose. But notice which duplicates survived: id 3 (“What HTTP status code does Meridian return…”) is clearly the same intent as id 1, and id 5 (“how many requests per minute does pro allow??”) is the same as id 4 — yet both slipped through. Token-set Jaccard at a 0.6 threshold only caught the closest pairs (it collapsed id 2 into id 1 and id 14 into id 6). Simple normalization is a useful floor, not a ceiling; near-duplicate clustering is genuinely hard, and a stricter threshold or an embedding-based similarity would catch more. The demo shows you a real, imperfect first pass rather than pretending the problem is solved.

Promote a failure into a regression case#

The candidate set still is not a test set, because the questions have no reference answers. Labeling is the work — and the labels you want most are for the failures. Look at id 7: a user asked whether they could change a database’s region, and Docent confidently answered “You can switch it anytime from the dashboard settings.” That is wrong. Meridian’s regions doc says the region is fixed at creation. The user’s thumbs-down was correct, and this is precisely the kind of bug that a golden set exists to freeze.

Promoting it means writing the correct reference (drawn from the canonical regions page), recording the doc id, tagging it, and keeping a note about the original failure so future readers know why this case is in the set:

import warnings, json
warnings.filterwarnings("ignore")

# One thumbs-down entry pulled straight from the production log (id=7). A user
# asked whether they could change a database's region, and Docent confidently
# said yes -- which is wrong. The Meridian regions doc says the region is fixed
# at creation. This is exactly the kind of failure a golden set should freeze.
failure = {
    "id": 7,
    "question": "Can I change a database region after creating it?",
    "answer": "You can switch it anytime from the dashboard settings.",
    "retrieved": ["regions"],
    "feedback": "down",
}

# Labeling: the hardest step. A human (or a careful reviewer) writes the
# known-correct reference answer and records which doc it comes from. This one
# is drawn from the canonical Meridian regions page.
regression_case = {
    "id": "reg-regions-fixed-001",
    "question": failure["question"],
    "reference": ("No; the region is fixed at creation and you must export and "
                  "re-import into a new database to move regions."),
    "doc_id": "regions",
    "tags": ["regions", "regression", "was_thumbs_down"],
    "source": "production-log",
    "note": "Docent previously answered 'you can switch it anytime' -- a factual error. "
            "Frozen so the next model change cannot silently reintroduce it.",
}

print("Before (raw failure from the log):")
print(f"  question: {failure['question']}")
print(f"  answer:   {failure['answer']}")
print(f"  feedback: {failure['feedback']}")
print()
print("After (promoted into a labeled regression case in the golden set):")
print(json.dumps(regression_case, indent=2))

Running it shows the raw failure on one side and the frozen regression case on the other:

Before (raw failure from the log):
  question: Can I change a database region after creating it?
  answer:   You can switch it anytime from the dashboard settings.
  feedback: down

After (promoted into a labeled regression case in the golden set):
{
  "id": "reg-regions-fixed-001",
  "question": "Can I change a database region after creating it?",
  "reference": "No; the region is fixed at creation and you must export and re-import into a new database to move regions.",
  "doc_id": "regions",
  "tags": [
    "regions",
    "regression",
    "was_thumbs_down"
  ],
  "source": "production-log",
  "note": "Docent previously answered 'you can switch it anytime' -- a factual error. Frozen so the next model change cannot silently reintroduce it."
}

That single object is worth more than a dozen questions you invented. It records a real bug, the correct answer, where the answer comes from, and why the case exists. From now on, every time you evaluate Docent — a prompt tweak, a model upgrade, a retrieval change — this case runs. If a future version of Docent ever again says you can switch regions “anytime,” the regression case fails and you catch it before a user does. That is the whole payoff of the pipeline: your failures stop being one-time embarrassments and become permanent tests.

Practice Exercises#

Exercise 1: Tune the dedupe threshold#

The demo left ids 3 and 5 in the “unique” set even though they are clearly restatements of ids 1 and 4. Suppose you lower near_duplicate’s threshold from 0.6 to 0.45. In words, what happens to the count of unique questions, and what is the risk of pushing the threshold too low — say, to 0.2?

Exercise 2: Add a targeted slow-and-costly rule#

Right now the sample targets only thumbs-down turns. Extend the sampling step so it also always keeps any turn whose latency_s is above 2.2 or whose cost_usd is above 0.0009, even if the user gave no feedback. Using the PROD_LOG in the lesson, which additional ids would that rule pull in?

Exercise 3: Write the note field for a second failure#

Id 11 (“Does Meridian support MongoDB-style aggregation pipelines?”) is a thumbs-down where Docent hallucinated a pipeline() helper that does not exist. Write the reference and note fields for its regression case. Remember what the correct behavior is for a question the Meridian docs cannot answer.

Summary#

Hand-written test sets measure the users you imagine; production logs measure the users you have. This lesson walked the pipeline that turns raw Docent traffic into a trustworthy golden set. You capture each turn as a structured record — question, answer, retrieved docs, feedback, latency, cost. You deduplicate near-identical phrasings, and you saw honestly that a simple token-set Jaccard at 0.6 catches the closest pairs but lets other restatements slip through, so clustering is a floor to improve on, not a solved problem. You sample with intent: target every thumbs-down failure, then add a stratified slice of the rest. You label the survivors with reference answers — the hardest, most human step, because the log gives you the question but never the answer. And you promote the failures into frozen regression cases, so a bug that reached a real user can never silently return. The real, deterministic run took 15 logged → 13 unique → 8 promoted candidates, identical on a second run, and you turned one thumbs-down about database regions into a fully labeled regression case. Running underneath all of it: scrub secrets and PII before a log line ever becomes a fixture, because a test set is a file the whole team can read.

Key Concepts#

• Production log — a structured record of each real Docent turn (question, answer, retrieved doc ids, user feedback, latency, cost) that captures how users actually phrase and combine questions.
• Capture — logging the fields you will need later, including feedback and operational numbers, not just the answer text.
• Deduplicate / cluster — collapsing near-identical questions so the test set does not over-weight popular phrasings; here, token-set Jaccard over normalized text, an imperfect first pass.
• Sampling (random + stratified + targeted) — choosing which unique questions to promote by mixing a fair cross-section, deliberate topic coverage, and every known failure.
• Labeling — writing the correct reference answer and recording its source doc; the hardest step, because the log never contains the reference.
• Promotion — moving labeled cases into the golden set, with past failures kept as regression cases.
• Regression case — a frozen test built from a thumbs-down failure, with the correct reference, so the same bug cannot be silently reintroduced by a later change.
• PII / secret scrubbing — redacting keys, emails, and personal data as you capture, because a fixture in your repo is effectively public.

Why This Matters#

The gap between “our eval passes” and “our users are happy” is usually the gap between the test set you imagined and the traffic you actually get. Teams that grow their golden set from logs close that gap continuously: every real failure becomes a permanent test, every new way of phrasing a question eventually gets covered, and the dataset tracks the product instead of drifting behind it. The regression discipline is the highest-leverage part — most painful LLM incidents are old bugs returning after a prompt tweak or model upgrade that “looked fine,” and a frozen regression case is exactly what turns that silent reappearance into a caught failure. Do the unglamorous work — capture the right fields, scrub the PII, label the failures — and your evaluation stops being a snapshot of your assumptions and becomes a living mirror of your real users. Next, you will make sure that mirror has no blind spots: measuring coverage, slicing the dataset, and keeping it clean.

Continue Building Your Skills#

You can now take a stream of raw Docent traffic and turn it into a test set worth trusting: capture the right fields, deduplicate the near-copies, target the thumbs-down failures, label the survivors, and freeze the failures as regression cases — all while scrubbing PII before anything becomes a fixture. The real pipeline you ran took 15 logged turns to 13 unique to 8 labeled candidates, deterministically, and promoted one region-change failure into a permanent regression test. Before moving on, try extending the demo with Exercise 2’s operational-outlier rule and Exercise 3’s refusal regression case, then imagine running the funnel on a thousand log lines instead of fifteen — the same five stages scale, and the labeling step is where your real effort will go. In Lesson 4 you will make sure the set those stages produce actually covers Meridian’s topics and difficulty levels, so no slice of your users hides in a blind spot.