Lesson 5 - Guided Project: Build Docent's Golden Dataset

Welcome to the Guided Project#

This is the capstone for Building Eval Datasets, and unlike the earlier lessons it produces something permanent: the actual golden dataset that every later module in this course will score against. Up to now you learned the properties of a good golden set, wrote test cases by hand, mined logs, and measured coverage. Here you put all of it together and assemble Docent’s dataset for real — schema first, then cases, then an honest coverage audit, then a serialized artifact you can load anywhere.

The four stages mirror how a careful team actually builds an eval set. You define a schema so every case is shaped and validated the same way; you author the cases deliberately, covering all eight Meridian doc pages and mixing in paraphrases, a refusal probe, and an adversarial false-premise trap; you audit coverage to prove there are no blind spots or duplicates; and you persist the whole thing to JSONL and read it back to confirm the round-trip is lossless. Almost everything here is deterministic — the same code produces the same dataset and the same report every single run — which is exactly the property a shared artifact needs. Let’s build it.

By the end of this project, you will be able to:

• Define a strict TestCase schema with pydantic that validates every field and rejects malformed cases on construction
• Author a diverse golden set covering all eight Meridian doc pages, including paraphrase pairs, an out-of-scope refusal, and an adversarial false-premise case
• Audit a dataset for coverage gaps, per-category and per-difficulty balance, and duplicates before you trust it
• Serialize the dataset to JSONL and validate a lossless round-trip, producing the reusable artifact later modules depend on

Stage 1: Define the Schema#

A golden dataset is only trustworthy if every case is shaped the same way and can’t quietly drift. A free-form list of dicts lets a typo — a misspelled category, a missing reference, a source_doc that points at a page you never wrote — slip in unnoticed and corrupt your metrics later. So we start with a schema: a pydantic TestCase model that names every field, constrains its type, and refuses to build an invalid case at all.

The seven fields are the shape we’ll reuse for the rest of the course: id (a stable handle like tc-03), input (the question we ask Docent), reference (the known-correct short answer), category and difficulty and type (the tags we slice on), and source_doc (which Meridian page the answer comes from, or None for out-of-scope cases). Using Literal for the tag fields means an unknown value is a validation error, not a silent typo, and a field_validator checks that any source_doc is a real page id.

from typing import Optional, Literal
from pydantic import BaseModel, Field, field_validator, ValidationError

DOC_IDS = {"auth", "ratelimits", "plans", "regions",
           "errors", "backups", "sdks", "deletion"}

class TestCase(BaseModel):
    id: str = Field(pattern=r"^tc-\d{2}$")
    input: str = Field(min_length=8)
    reference: str = Field(min_length=1)
    category: Literal["factual", "procedural", "comparison", "refusal", "adversarial"]
    difficulty: Literal["easy", "medium", "hard"]
    type: Literal["lookup", "paraphrase", "out_of_scope", "false_premise"]
    source_doc: Optional[str] = None

    @field_validator("source_doc")
    @classmethod
    def known_doc(cls, v):
        if v is not None and v not in DOC_IDS:
            raise ValueError(f"unknown source_doc: {v!r}")
        return v

# a valid case builds cleanly
example = TestCase(
    id="tc-01",
    input="What HTTP status code does Meridian return when you exceed the rate limit?",
    reference="429",
    category="factual", difficulty="easy", type="lookup", source_doc="ratelimits",
)
print("schema fields:", list(TestCase.model_fields.keys()))
print("example valid:", example.id, "->", example.reference, f"({example.source_doc})")

# a broken case is rejected loudly — bad input length AND an unknown source_doc
try:
    TestCase(id="tc-99", input="?", reference="x", category="factual",
             difficulty="easy", type="lookup", source_doc="billing")
except ValidationError as e:
    print("rejected bad case:", len(e.errors()), "error(s) ->",
          ", ".join(err["loc"][0] for err in e.errors()))

schema fields: ['id', 'input', 'reference', 'category', 'difficulty', 'type', 'source_doc']
example valid: tc-01 -> 429 (ratelimits)
rejected bad case: 2 error(s) -> input, source_doc

The schema does exactly what a schema should: it accepts a well-formed case and it refuses a malformed one, telling you precisely which fields failed (here, the eight-character-minimum on input and the unknown source_doc="billing"). Because validation runs the moment you construct a TestCase, every case in Stage 2 is guaranteed valid by the time it lands in the list — you can’t build a broken dataset even by accident.

Stage 2: Author the Cases#

Now we write the dataset itself. This is the part that takes judgment, not code: a golden set is only as good as the questions in it, and a pile of easy factual lookups will make Docent look flawless while hiding every weakness that matters. So we author deliberately for diversity across three axes — the eight doc pages (coverage), the categories and difficulties (balance), and the kinds of challenge (lookup, paraphrase, refusal, adversarial).

Fourteen cases, each drawn from Meridian’s docs and kept faithful to them:

• Every one of the eight doc pages appears at least once, so no part of the product is untested.
• Two paraphrase pairs (tc-01/tc-02 on auth, tc-03/tc-04 on rate limits) ask the same underlying question two different ways — a system that only handles one phrasing has a robustness gap a single question would never expose.
• One out-of-scope refusal (tc-13, the MongoDB question) the docs simply can’t answer; the correct behavior is to say so.
• One adversarial false-premise case (tc-14) that smuggles in a fact the docs never state (“Meridian bills per gigabyte of egress”) — a good assistant rejects the premise instead of playing along.

RAW = [
    # --- auth (page 1): a paraphrase pair ---
    dict(id="tc-01", input="How does Meridian authenticate an API request?",
         reference="With an API key sent as a bearer token in the Authorization header.",
         category="procedural", difficulty="medium", type="lookup", source_doc="auth"),
    dict(id="tc-02", input="What do I put in the Authorization header to call the Meridian API?",
         reference="An API key, sent as a bearer token.",
         category="procedural", difficulty="medium", type="paraphrase", source_doc="auth"),
    # --- ratelimits (page 2): another paraphrase pair ---
    dict(id="tc-03", input="What HTTP status code does Meridian return when you exceed the rate limit?",
         reference="429", category="factual", difficulty="easy", type="lookup", source_doc="ratelimits"),
    dict(id="tc-04", input="If I send requests too fast, which HTTP error does Meridian give me?",
         reference="429", category="factual", difficulty="easy", type="paraphrase", source_doc="ratelimits"),
    dict(id="tc-05", input="How many requests per minute does the Pro plan allow?",
         reference="600 requests per minute", category="factual", difficulty="easy",
         type="lookup", source_doc="ratelimits"),
    # --- plans (page 3) ---
    dict(id="tc-06", input="How much does the Pro plan cost per month?",
         reference="25 dollars per month", category="factual", difficulty="easy",
         type="lookup", source_doc="plans"),
    # --- regions (page 4) ---
    dict(id="tc-07", input="Can you change a database's region after it is created?",
         reference="No; the region is fixed at creation and you must export and re-import into a new database to move regions.",
         category="procedural", difficulty="medium", type="lookup", source_doc="regions"),
    # --- errors (page 5) ---
    dict(id="tc-08", input="What does a 401 error mean?",
         reference="A missing or invalid API key", category="factual", difficulty="easy",
         type="lookup", source_doc="errors"),
    # --- backups (page 6): one lookup + one comparison ---
    dict(id="tc-09", input="How long are backups retained on the Scale plan?",
         reference="90 days", category="factual", difficulty="easy", type="lookup", source_doc="backups"),
    dict(id="tc-10", input="Which plans support point-in-time recovery?",
         reference="Only the Scale plan", category="comparison", difficulty="medium",
         type="lookup", source_doc="backups"),
    # --- sdks (page 7) ---
    dict(id="tc-11", input="Which environment variable does the Python SDK read for the API key?",
         reference="MERIDIAN_API_KEY", category="factual", difficulty="easy",
         type="lookup", source_doc="sdks"),
    # --- deletion (page 8) ---
    dict(id="tc-12", input="How long is the grace period before a deleted database is permanently purged?",
         reference="24 hours", category="factual", difficulty="medium", type="lookup", source_doc="deletion"),
    # --- out-of-scope refusal ---
    dict(id="tc-13", input="Does Meridian support MongoDB-style aggregation pipelines?",
         reference="Not covered in the docs; Docent should say it doesn't have that.",
         category="refusal", difficulty="hard", type="out_of_scope", source_doc=None),
    # --- adversarial false premise (docs never mention egress billing) ---
    dict(id="tc-14", input="Since Meridian bills per gigabyte of egress, how do I lower my egress charges?",
         reference="Reject the premise: the docs never mention egress billing, so Docent should refuse.",
         category="adversarial", difficulty="hard", type="false_premise", source_doc=None),
]

GOLDEN = [TestCase(**row) for row in RAW]  # every case validated on construction
print("authored & validated:", len(GOLDEN), "cases")
print("sample:", GOLDEN[3].id, "|", GOLDEN[3].type, "|", GOLDEN[3].input[:38], "...")
print("refusal cases:", [c.id for c in GOLDEN if c.category in ("refusal", "adversarial")])

authored & validated: 14 cases
sample: tc-04 | paraphrase | If I send requests too fast, which HTT ...
refusal cases: ['tc-13', 'tc-14']

Fourteen cases, every one validated the instant it was constructed — if any field had been wrong, [TestCase(**row) for row in RAW] would have raised on that row instead of building the list. The two “refusal” cases (tc-13, tc-14) are the ones most teams forget to write, and they’re the ones that catch the most dangerous failure: an assistant that confidently invents an answer it has no basis for. We built them in from the start.

Stage 3: Audit Coverage#

Authoring cases is one thing; proving the set has no blind spots is another. A coverage audit is a short, deterministic report that answers the questions a reviewer would ask before trusting the dataset: is every doc page represented? Is the difficulty spread reasonable, or is it all easy lookups? Are there accidental duplicates inflating the count? We compute all of it from the objects themselves — no manual bookkeeping to fall out of date.

from collections import Counter

covered = {c.source_doc for c in GOLDEN if c.source_doc is not None}
missing = DOC_IDS - covered
by_cat = Counter(c.category for c in GOLDEN)
by_diff = Counter(c.difficulty for c in GOLDEN)
by_type = Counter(c.type for c in GOLDEN)

print(f"doc pages covered: {len(covered)}/{len(DOC_IDS)}")
print("  missing pages:", sorted(missing) or "none")
print("by category:  ", dict(sorted(by_cat.items())))
print("by difficulty:", dict(sorted(by_diff.items())))
print("by type:      ", dict(sorted(by_type.items())))

# dedup checks: ids must be unique, and no two inputs identical
ids = [c.id for c in GOLDEN]
inputs = [c.input.strip().lower() for c in GOLDEN]
print("duplicate ids:   ", [i for i, n in Counter(ids).items() if n > 1] or "none")
print("duplicate inputs:", [q for q, n in Counter(inputs).items() if n > 1] or "none")

# paraphrase pairs share a source_doc but ask differently — expected, NOT a duplicate
pairs = [(a.id, b.id) for a in GOLDEN for b in GOLDEN
         if a.id < b.id and a.type == "lookup" and b.type == "paraphrase"
         and a.source_doc == b.source_doc]
print("paraphrase pairs:", pairs)

ok = (len(missing) == 0
      and not [i for i, n in Counter(ids).items() if n > 1]
      and not [q for q, n in Counter(inputs).items() if n > 1])
print("coverage audit passed:", ok)

doc pages covered: 8/8
  missing pages: none
by category:   {'adversarial': 1, 'comparison': 1, 'factual': 8, 'procedural': 3, 'refusal': 1}
by difficulty: {'easy': 7, 'hard': 2, 'medium': 5}
by type:       {'false_premise': 1, 'lookup': 10, 'out_of_scope': 1, 'paraphrase': 2}
duplicate ids:    none
duplicate inputs: none
paraphrase pairs: [('tc-01', 'tc-02'), ('tc-03', 'tc-04')]
coverage audit passed: True

The report is clean: 8 of 8 doc pages covered, no missing pages, no duplicate ids, and no duplicate inputs. The category and difficulty breakdowns show a deliberate spread — mostly easy factual lookups (the bread and butter), leavened with medium procedural questions and two hard refusal cases. Note the dedup check is smart enough not to flag the paraphrase pairs: tc-01/tc-02 and tc-03/tc-04 share a source_doc on purpose, but their input text differs, so they’re distinct cases, not duplicates. This whole stage touches no model and no clock — run it again and you get byte-identical output, which is what lets you drop it into CI as a gate.

Stage 4: Persist It#

A dataset that lives only in memory helps no one — the whole point is a reusable artifact that Module 3’s metrics, Module 4’s slices, and the observability modules all load from the same file. We serialize to JSONL (one JSON object per line), the format eval tooling expects because it streams, diffs cleanly in git, and appends without rewriting. Crucially, we write it to a temporary directory, not the repo: this lesson demonstrates the mechanics, and you don’t want throwaway files committed. In your own project you’d write to a versioned path like data/golden_v1.jsonl.

The real test of serialization is the round-trip: write the set, read it back, re-validate every line against the same schema, and confirm nothing changed. If the loaded objects match the originals field for field, the file is a faithful snapshot you can trust downstream.

import json, tempfile, os

# write to a TEMP dir, never the repo — this is a demo of the mechanics
tmpdir = tempfile.mkdtemp(prefix="docent-golden-")
path = os.path.join(tmpdir, "golden_v1.jsonl")

with open(path, "w") as f:
    for c in GOLDEN:
        f.write(c.model_dump_json() + "\n")

size = os.path.getsize(path)
print("wrote:", os.path.basename(path), f"({size} bytes) to a temp dir")

# read it back and re-validate every line against the schema
loaded = []
with open(path) as f:
    for line in f:
        loaded.append(TestCase(**json.loads(line)))

print("read back:", len(loaded), "cases")
print("count matches:", len(loaded) == len(GOLDEN))
print("round-trip identical:", [c.model_dump() for c in loaded] == [c.model_dump() for c in GOLDEN])
print("re-validated coverage:", len({c.source_doc for c in loaded if c.source_doc}), "/ 8 pages")

# clean up the temp artifact so nothing lingers
os.remove(path); os.rmdir(tmpdir)
print("temp artifact removed:", not os.path.exists(path))

wrote: golden_v1.jsonl (3015 bytes) to a temp dir
read back: 14 cases
count matches: True
round-trip identical: True
re-validated coverage: 8 / 8 pages
temp artifact removed: True

The round-trip is lossless: 14 cases out, 14 cases back, every field identical (round-trip identical: True), and the reloaded set re-validates cleanly against TestCase with all 8 pages still covered. That last check matters more than it looks — reading a JSONL line back through TestCase(**json.loads(line)) means a corrupted or hand-edited file would fail the same validation Stage 1 defined, so your loader is also your integrity check. This is the artifact the rest of the course builds on. You just assembled Docent’s golden dataset end to end, and every metric you learn from here forward will score against exactly these 14 cases.

Practice Exercises#

Exercise 1: Add a tags field to the schema#

The current schema has one slot per axis, but real cases often need free-form labels — "beta-feature", "customer-reported", "regression". Add an optional tags: list[str] field to TestCase (defaulting to an empty list), tag two or three of the existing cases, and extend the Stage 3 audit to print a count per tag.

Exercise 2: Turn the audit into a hard gate#

Right now the audit prints whether it passed. Wrap the coverage and dedup checks in a function audit(dataset) that raises AssertionError with a clear message when any page is missing or a duplicate exists, and returns the report dict otherwise. Then deliberately delete the only sdks case and confirm the gate fails loudly.

Exercise 3: Version the artifact with a manifest#

A bare JSONL file doesn’t record when it was built or how many cases it holds. Alongside golden_v1.jsonl, write a small manifest.json (still to the temp dir) containing the version string, the case count, the list of covered doc pages, and a build date. On load, read the manifest first and assert the JSONL line count matches its count.

Summary#

You built Docent’s golden dataset end to end — the artifact the rest of this course scores against. You defined a strict pydantic schema (TestCase) with seven fields that validates every case on construction and rejects malformed ones with named errors; you authored 14 cases covering all eight Meridian doc pages, deliberately mixing easy factual lookups with procedural and comparison questions, two paraphrase pairs, an out-of-scope refusal, and an adversarial false-premise trap; you audited coverage and got a clean report — 8/8 pages, no duplicate ids or inputs, a sensible category and difficulty spread — that runs byte-identical every time; and you persisted the set to JSONL in a temp directory and proved a lossless round-trip by reading it back and re-validating it against the same schema. Almost every stage is deterministic, which is exactly what a shared, versioned dataset needs to be.

Key Concepts#

• Schema-first datasets — a pydantic model that validates each case at construction time, so a malformed case fails next to the line that caused it instead of corrupting metrics later.
• Deliberate diversity — coverage of every doc page, a balanced spread of categories and difficulties, plus paraphrase pairs and refusal/adversarial cases that expose failures a pile of easy lookups would hide.
• Coverage audit — a deterministic report (pages covered, per-tag counts, dedup checks) that proves the set has no blind spots before you trust it, and can run as a CI gate.
• Lossless round-trip — serialize to JSONL, read back, and re-validate against the schema; if the reloaded objects match field for field, the file is a faithful, reusable snapshot.
• Reusable artifact — one versioned golden_v1.jsonl that every later module loads, so results are comparable and a coverage regression surfaces as a failing audit, not silent metric drift.

Why This Matters#

Every metric in the modules ahead — deterministic scorers, semantic similarity, LLM-as-judge, slice analysis, production monitoring — reads from a dataset, and none of them can be more trustworthy than that dataset is. By building the golden set schema-first, auditing it for coverage, and proving the serialization round-trips losslessly, you’ve produced something a team can actually depend on: a versioned artifact where every case is validated, every doc page is represented, and the whole thing is reproducible enough to gate a build on. That discipline is the difference between an eval that catches a real regression before it ships and a dashboard of confident numbers that quietly mean nothing.

Continue Building Your Skills#

You’ve assembled Docent’s golden dataset — schema-validated, covering all eight doc pages, audited for gaps and duplicates, and serialized to a JSONL artifact that round-trips losslessly. That file is the foundation everything else stands on: it doesn’t change when you swap scorers, so any movement in your numbers reflects the system, not the questions. Next you’ll put it to work. In Module 3 you’ll write the deterministic, reference-based metrics — exact match, token F1, ROUGE — that score Docent’s answers against exactly these 14 references, turning your carefully-built dataset into the numbers you’ll track for the rest of the course.