Lesson 1 - Golden Sets: What Makes a Good Eval Dataset

Welcome to Golden Sets#

In Module 1 you caught Docent – our assistant for the Meridian serverless database docs – failing on a question its own documentation answered, and you saw the probability math proving that eyeballing three answers is not measurement. The cure was to run many labeled questions and count the passes. This module is about building that set of labeled questions well, because every number you will ever compute about Docent – accuracy, refusal rate, faithfulness – is only as trustworthy as the dataset you compute it on.

That dataset has a name: a golden set. It’s a collection of example inputs, each paired with a known-correct reference answer or a clear pass criterion. “Golden” means you trust the labels: when Docent’s output disagrees with a golden reference, the reference is right and Docent is wrong, by definition. Get the golden set wrong and you don’t get slightly-off scores – you get confident scores that measure the wrong thing, which is worse than no score at all.

This lesson defines what “good” means for a golden set, shows what a strong one for Docent looks like, and ends with a real audit that scores the health of a dataset before you ever trust it.

In this lesson, you will:

• Learn the five properties of a trustworthy golden set: representative, labeled, diverse, stocked with hard cases and negatives, and partly held-out
• See how size trades off against quality, and how big “enough” really is
• Understand provenance and labeling quality: who wrote the reference, and why you version labels
• Recognize a bad golden set on sight, and run a real dataset-health audit over Docent’s canonical Meridian Q&A set

The five properties of a golden set#

A golden set is not just “a bunch of questions.” A good one has five properties, and each one exists to close a specific way that an eval score can lie to you.

1. Representative. The set should match the distribution of questions your users actually ask, not the ones that are easy to write. If real Meridian users mostly ask about rate limits, errors, and pricing, but your golden set is 80% questions about backup retention windows, then a great score tells you Docent is good at something users rarely do. The distribution matters as much as the content: an eval that over-samples the easy region of question-space inflates every metric.

2. Labeled. Every example needs a reference answer (the known-correct short answer) or, when there’s no single right string, a clear pass criterion (“must mention 429”, “must refuse”). Without a label there is nothing to grade against, and grading collapses back into the eyeballing you’re trying to escape. For Docent, "How much does the Pro plan cost per month?" carries the reference "25 dollars per month" – an unambiguous target.

3. Diverse. Cover categories, phrasings, and difficulty. Categories: factual lookups, comparisons, refusals. Phrasings: the same fact asked three different ways, because users don’t all type the same words. Difficulty: trivially easy lookups and multi-fact questions that require reading two sentences of a doc page. A set that’s all one category or one phrasing measures one narrow skill and calls it “quality.”

4. Hard cases and negatives. A golden set that only contains answerable, happy-path questions can never catch Docent’s most dangerous failure – confidently answering something it shouldn’t. You need negatives: out-of-scope questions where the correct behavior is to refuse (Meridian’s docs say nothing about MongoDB aggregation, so "I don't have that in the docs" is the right answer), adversarial phrasings that try to bait a wrong answer, and genuinely ambiguous questions. These are the examples that separate a good assistant from a lucky one.

5. Held-out. Keep a portion of the set in reserve that you never look at while tuning prompts or retrieval. If you tweak Docent until it aces every question you can see, your score measures memorization, not generalization – you’ve overfit the eval. A held-out split that you score only occasionally keeps the number honest. This is the same train/test discipline from machine learning, applied to prompts instead of model weights.

What a strong golden set for Docent needs#

Concretely, a trustworthy Docent golden set has to exercise the whole product, not a corner of it. Meridian’s documentation is 8 pages – auth, ratelimits, plans, regions, errors, backups, sdks, deletion – so a representative, diverse set needs at minimum:

• Factual lookups spread across all 8 pages. If a page has zero questions, you have zero evidence about how Docent handles it. A page with no coverage is a blind spot, and blind spots are exactly where the Module 1 failure lived.
• Paraphrased questions. The same fact asked more than one way – "How much does the Pro plan cost per month?" and "What's the monthly price of Pro?" – so a good score can’t come from Docent memorizing one exact phrasing.
• The out-of-scope refusal case. At least one question the docs genuinely cannot answer (the MongoDB aggregation probe), whose correct label is a refusal. This is the negative that catches over-confident answering.
• An ambiguous case. A question that could mean two things ("How long are backups kept?" – on which plan?), where a good answer either asks for clarification or states the assumption. Ambiguity is where real users live.

The canonical set you’ll audit in a moment has 11 examples: 10 factual references drawn from the docs plus 1 out-of-scope probe. It’s deliberately small – and that’s a feature, which brings us to size.

Size versus quality: how big is “enough”#

The instinct is that bigger is better. It isn’t. A small, well-chosen set beats a big noisy one, for a simple reason: a noisy label doesn’t just add nothing, it actively corrupts the score. If 10% of a 1,000-example set has wrong references, then 100 examples are silently grading Docent against the wrong answer, and no amount of size averages that away – it just makes the wrong number look precise.

So aim for clean and representative first, large second. Some rules of thumb for a golden set like Docent’s:

• A few dozen carefully labeled examples is enough to catch gross regressions and gives you a real, if coarse, accuracy number. This is where you start – and it’s already infinitely better than three spot-checks.
• Per-slice minimums matter more than the total. An overall accuracy of 92% is meaningless if the deletion slice has 2 examples. As a rough floor, aim for at least 5 examples per category or doc page you want to report a number on – fewer than that and one flipped answer swings the slice by 20 points or more, so the metric is noise.
• Hundreds of examples let you detect small, real differences between two versions of Docent with statistical confidence. You grow toward this deliberately, adding examples from real failures (a technique later lessons cover), not by bulk-generating filler.

The math of per-slice minimums is worth internalizing: with $n$ examples in a slice, one wrong answer moves that slice’s accuracy by $1/n$. At $n = 2$ that’s 50 points of swing per answer; at $n = 20$ it’s 5. Small slices don’t measure – they gossip.

Provenance and labeling quality#

A label is only as trustworthy as the process that produced it. Two questions decide whether you can trust a golden set’s references:

• Who wrote the reference, and from what? A reference answer written by a Meridian domain expert reading the actual docs is trustworthy. One written by asking another LLM “what’s the answer?” and pasting it in is not a golden label – it’s a second model’s guess, and evaluating one model against another model’s unverified output measures agreement, not correctness. For Docent, every reference is traceable to a specific doc page ("25 dollars per month" comes from the plans page), so any human can verify it in seconds.
• Do independent labelers agree? When answers are judgment calls (is this refusal appropriate? is this summary faithful?), have more than one person label the same examples and measure how often they agree – inter-annotator agreement. Low agreement is a red flag that the task is underspecified: if two careful humans can’t agree on the right answer, no metric built on those labels means anything. High agreement means the label is a real target.

And whatever the labels are, version them. Treat the golden set like code: it lives in a file under source control, changes go through review, and every change bumps a version number. When Docent’s score jumps between two runs, the first question is “did the model change, or did the dataset change?” – and you can only answer it if the dataset has versions. A golden set that’s edited in place, with no history, makes every score unreproducible.

What a bad golden set looks like#

It’s easier to trust the five properties once you’ve seen them violated. A bad golden set usually has one or more of these tells:

• Only easy questions. Every example is a one-fact lookup with an exact keyword match. Docent scores 100%, ships, and falls over on the first multi-fact or paraphrased question a real user asks. Easy-only sets measure the easy region and extrapolate a lie.
• Only happy-path. No negatives, no out-of-scope probes. The set literally cannot detect over-confident answering, which is the failure that hurts users most. Every question has an answer, so a bot that never refuses looks perfect.
• Labels that encode one phrasing. The reference for a plan-price question is the exact string "The Pro plan costs 25 dollars per month.", and the grader marks anything else wrong – so a correct answer of "$25/mo" fails. The label is testing wording, not the fact. Good labels capture the fact to check for, not one sentence.
• Leakage from the prompt. The golden question is copied verbatim from an example already baked into Docent’s system prompt, so Docent “passes” by echoing text it was handed. That’s the eval equivalent of putting the answers on the test – the score is real and completely meaningless.

Notice that a bad golden set doesn’t produce low scores. It produces high, confident, wrong scores – which is exactly why you audit the dataset itself before you trust anything it tells you.

A real dataset-health audit#

Talk is cheap; let’s measure. Below is Docent’s canonical golden set as a plain Python list of dicts – 10 factual examples plus 1 out-of-scope probe – and a small audit function that computes health stats: how many examples, how many of Meridian’s 8 doc pages are covered, how many negatives, whether any questions are duplicated, and whether any factual example is missing its label. No model calls, no randomness – just arithmetic over the data, so it returns the exact same numbers every run.

DOC_IDS = {"auth", "ratelimits", "plans", "regions", "errors", "backups", "sdks", "deletion"}

GOLDEN = [
    {"q": "What HTTP status code does Meridian return when you exceed the rate limit?",
     "ref": "429", "doc": "ratelimits", "kind": "factual"},
    {"q": "How many requests per minute does the Pro plan allow?",
     "ref": "600 requests per minute", "doc": "ratelimits", "kind": "factual"},
    {"q": "How much does the Pro plan cost per month?",
     "ref": "25 dollars per month", "doc": "plans", "kind": "factual"},
    {"q": "Can you change a database's region after it is created?",
     "ref": "No; the region is fixed at creation and you must export and re-import into a new database to move regions.",
     "doc": "regions", "kind": "factual"},
    {"q": "How long are backups retained on the Scale plan?",
     "ref": "90 days", "doc": "backups", "kind": "factual"},
    {"q": "Which environment variable does the Python SDK read for the API key?",
     "ref": "MERIDIAN_API_KEY", "doc": "sdks", "kind": "factual"},
    {"q": "What does a 401 error mean?",
     "ref": "A missing or invalid API key", "doc": "errors", "kind": "factual"},
    {"q": "How long is the grace period before a deleted database is permanently purged?",
     "ref": "24 hours", "doc": "deletion", "kind": "factual"},
    {"q": "Which plans support point-in-time recovery?",
     "ref": "Only the Scale plan", "doc": "backups", "kind": "factual"},
    {"q": "What does Meridian charge for the Free plan?",
     "ref": "0 dollars per month (it is free), including 1 GB of storage", "doc": "plans", "kind": "factual"},
    {"q": "Does Meridian support MongoDB-style aggregation pipelines?",
     "ref": "(not covered in the docs -- Docent should say it doesn't know)", "doc": None, "kind": "out_of_scope"},
]

def audit(dataset, doc_ids):
    # coverage: which in-scope doc pages are exercised by at least one question
    covered = {ex["doc"] for ex in dataset if ex["doc"] is not None}
    uncovered = doc_ids - covered

    # duplicate detection on whitespace/case-normalized question text
    seen, duplicates = {}, []
    for i, ex in enumerate(dataset):
        key = " ".join(ex["q"].lower().split())
        if key in seen:
            duplicates.append((seen[key], i))
        else:
            seen[key] = i

    # missing labels: every factual example must carry a non-empty reference
    missing_ref = [i for i, ex in enumerate(dataset)
                   if ex["kind"] == "factual" and not str(ex.get("ref", "")).strip()]

    return {
        "total": len(dataset),
        "docs_covered": len(covered),
        "docs_total": len(doc_ids),
        "uncovered_docs": sorted(uncovered),
        "out_of_scope": sum(1 for ex in dataset if ex["kind"] == "out_of_scope"),
        "duplicates": len(duplicates),
        "missing_ref": len(missing_ref),
    }

r = audit(GOLDEN, DOC_IDS)
print("Golden set health report")
print("-" * 40)
print(f"examples ................. {r['total']}")
print(f"doc pages covered ........ {r['docs_covered']} / {r['docs_total']}")
print(f"uncovered doc pages ...... {r['uncovered_docs']}")
print(f"out-of-scope / negatives . {r['out_of_scope']}")
print(f"duplicate questions ...... {r['duplicates']}")
print(f"examples missing a label . {r['missing_ref']}")
print(f"coverage ................. {100 * r['docs_covered'] / r['docs_total']:.0f}%")

Golden set health report
----------------------------------------
examples ................. 11
doc pages covered ........ 7 / 8
uncovered doc pages ...... ['auth']
out-of-scope / negatives . 1
duplicate questions ...... 0
examples missing a label . 0
coverage ................. 88%

The audit found something a glance at the list would miss: the set covers 7 of 8 doc pages, and the one it skips is auth. Docent could be silently broken on every authentication question – exactly the Module 1 failure mode – and this golden set would never catch it, because it asks nothing about auth. That’s a concrete, actionable gap: write at least one auth question with a reference ("A key is shown once at creation and cannot be recovered, only rotated.") and coverage goes to 8/8. The audit also confirms the healthy signs: exactly 1 negative present, no duplicates, and every factual example carries a label.

Because it’s pure arithmetic over the data, the report is fully reproducible – run it again and the numbers are identical:

r2 = audit(GOLDEN, DOC_IDS)
print("run 2 -> examples:", r2["total"],
      "| coverage:", f"{r2['docs_covered']}/{r2['docs_total']}",
      "| uncovered:", r2["uncovered_docs"],
      "| duplicates:", r2["duplicates"])

run 2 -> examples: 11 | coverage: 7/8 | uncovered: ['auth'] | duplicates: 0

Now watch the same audit catch the bad patterns from the previous section. Here’s a messier dataset: someone pasted the plan-price question a second time (with sloppy spacing and capitalization) and added a factual question with no reference answer. The audit flags both – the duplicate and the missing label – without anyone reading the list by hand:

messy = GOLDEN + [
    {"q": "How much   does the PRO plan cost per month?",   # accidental duplicate of an existing question
     "ref": "25 dollars per month", "doc": "plans", "kind": "factual"},
    {"q": "What time do automatic backups run?",            # oops: no reference answer written yet
     "ref": "", "doc": "backups", "kind": "factual"},
]

m = audit(messy, DOC_IDS)
print("Messy set health report")
print("-" * 40)
print(f"examples ................. {m['total']}")
print(f"doc pages covered ........ {m['docs_covered']} / {m['docs_total']}")
print(f"out-of-scope / negatives . {m['out_of_scope']}")
print(f"duplicate questions ...... {m['duplicates']}")
print(f"examples missing a label . {m['missing_ref']}")

Messy set health report
----------------------------------------
examples ................. 13
doc pages covered ........ 7 / 8
out-of-scope / negatives . 1
duplicate questions ...... 1
examples missing a label . 1

The duplicate is caught even though its spacing and capitalization differ from the original, because the check normalizes whitespace and case before comparing – and the unlabeled question is flagged before it can silently drag a score down. This is a preview of Lesson 4, where you’ll turn this kind of audit into a reusable quality gate you run every time the golden set changes. For now the point is simpler: you can measure the health of a dataset, and you should, before you trust a single number it produces.

Practice Exercises#

Exercise 1: Close the auth gap#

The audit reported auth as the one uncovered doc page. Write a new factual example – a question about Meridian authentication with a reference answer traceable to the auth doc text – append it to GOLDEN, and re-run audit. Confirm that coverage goes to 8/8 and uncovered doc pages becomes empty.

Exercise 2: Add a per-slice size check#

The current audit counts total examples but not how many land on each doc page, so it can’t warn you about a thinly-covered slice. Extend audit to also return a per_doc count (how many questions target each covered doc id) and print any doc that has fewer than 2 questions. Which pages in the canonical set would that flag?

Exercise 3: Spot the leakage#

Recall the “leakage from the prompt” anti-pattern: a golden question that copies text already in Docent’s system prompt. Look at the Docent implementation from Module 1 – its prompt instructs the model to say exactly "I don't have that in the docs." when the context lacks the answer. Explain in a sentence or two why using that exact string as the reference for the out-of-scope example is a mild form of leakage, and propose a pass criterion for refusals that doesn’t depend on matching that one sentence.

Summary#

A golden set is the labeled dataset every eval score is built on, and its quality is a hard ceiling on how much you can trust any number you compute from it. A good one is representative of real usage, labeled with a reference answer or clear pass criterion, diverse across categories and phrasings and difficulty, stocked with hard cases and negatives (out-of-scope, adversarial, ambiguous), and partly held-out so scores measure generalization, not memorization. Size helps only after quality: a small clean set beats a big noisy one, and per-slice minimums matter more than the total. Labels are only as good as their provenance – who wrote them and whether independent labelers agree – and they must be versioned like code. You then ran a real, deterministic audit over Docent’s canonical set and it found a concrete gap (auth uncovered, 7/8 pages, 88%), confirmed one negative and zero duplicates, and – on a deliberately messy variant – caught a whitespace-and-case duplicate and an unlabeled example automatically.

Key Concepts#

• Golden set – a labeled dataset of inputs paired with known-correct references or pass criteria; the authoritative ground truth an eval scores against.
• The five properties – representative, labeled, diverse, hard-cases-and-negatives, held-out; each closes a specific way a score can mislead you.
• Negatives / out-of-scope cases – examples whose correct answer is a refusal; the only way a golden set can detect over-confident answering.
• Size vs. quality – a small clean set beats a big noisy one; per-slice minimums (aim for 5+ per reported slice) matter more than the total, since one wrong answer swings an $n$-example slice by $1/n$.
• Provenance & versioning – trustworthy labels come from experts reading the source with measured inter-annotator agreement, and the dataset is version-controlled so scores are reproducible.
• Dataset audit – health stats (count, coverage, negatives, duplicates, missing labels) computed over the data itself, deterministically, before you trust any metric built on it.

Why This Matters#

Every eval you build for the rest of this course reads from a golden set, so a flaw in the dataset propagates silently into every metric, dashboard, and go/no-go decision downstream – and it does so as a high, confident, wrong number, which is more dangerous than no number. Teams routinely ship LLM features on the strength of an impressive eval score that turned out to be measuring an easy, happy-path, leaky slice of their real traffic. Knowing the five properties, respecting per-slice minimums, and auditing dataset health before trusting it is what separates an eval you can bet a release on from one that just makes you feel good. You now know what a good golden set is; next you’ll write one by hand, one deliberate test case at a time.

Continue Building Your Skills#

You now know what makes a golden set trustworthy – representative, labeled, diverse, negative-aware, and held-out – and you’ve run a real audit that turned “is this dataset any good?” into concrete numbers: 11 examples, 7/8 pages covered, one gap on auth, one negative, zero duplicates. That’s the standard your Docent dataset has to meet. The next step is to actually build it. In Lesson 2 you’ll write test cases by hand, making the deliberate choices each property demands – which questions to include, how to phrase the references so they test facts and not wording, and how to seed the hard cases and negatives that catch the failures a happy-path set never will.