https://datatweets.com/courses/llm-evaluation/building-eval-datasets/Recent content in Building Eval Datasets on DATATWEETSHugoenCopyright (c) 2026 DatatweetsTue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/building-eval-datasets/lesson-1-golden-sets/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/building-eval-datasets/lesson-1-golden-sets/A golden set is the labeled dataset every eval score is built on, so its quality caps the trust you can place in any number you compute. This lesson defines the five properties of a good one – representative, labeled, diverse, stocked with hard cases and negatives, and partly held-out – and shows what a strong Docent golden set needs across Meridian’s 8 doc pages. You’ll then run a real, deterministic dataset audit that counts examples, measures doc-page coverage, flags out-of-scope cases, and catches duplicates and missing labels, previewing the automated audit you’ll build in Lesson 4.https://datatweets.com/courses/llm-evaluation/building-eval-datasets/lesson-2-writing-test-cases-by-hand/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/building-eval-datasets/lesson-2-writing-test-cases-by-hand/A golden set is written one case at a time, and this lesson teaches the craft. You will learn the anatomy of a Docent test case — id, input, reference, category, difficulty, type, and source_doc — and the five case types to include deliberately: happy-path factual, paraphrases of the same fact, out-of-scope negatives that must refuse, ambiguous edge cases, and adversarial false-premise questions. You will write references a metric can actually score, then define a pydantic TestCase model, validate ten real Meridian cases, print a summary by type and category, and watch the schema reject a deliberately broken case — all deterministic, no API key needed.https://datatweets.com/courses/llm-evaluation/building-eval-datasets/lesson-3-from-production-logs-to-test-sets/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/building-eval-datasets/lesson-3-from-production-logs-to-test-sets/Hand-written test sets drift from how users really talk. This lesson shows how Docent’s golden set grows from production traffic instead: capture structured logs (question, answer, retrieved docs, feedback, latency, cost), deduplicate near-identical phrasings, sample the thumbs-down failures and a stratified slice of the rest, label the survivors with reference answers, and promote a past failure into a frozen regression case. You run a real, deterministic pipeline that takes 15 logged Meridian turns down to 13 unique to 8 labeled candidates, twice with identical numbers, and see why scrubbing PII comes first.https://datatweets.com/courses/llm-evaluation/building-eval-datasets/lesson-4-coverage-slicing-and-hygiene/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/building-eval-datasets/lesson-4-coverage-slicing-and-hygiene/A golden set you have not measured is a set you cannot trust. This lesson builds three deterministic checks for Docent’s Meridian test set: a coverage report that finds an untested doc page and confirms you have adversarial and hard cases, sliced accuracy that reveals an 81.8% average hiding a 33.3% ‘hard’ slice, and hygiene checks that flag a near-duplicate question and a held-out question leaking into new candidates. You will also run Docent live on the hard slice to see the current build actually handle it.https://datatweets.com/courses/llm-evaluation/building-eval-datasets/lesson-5-guided-project-docents-golden-dataset/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/building-eval-datasets/lesson-5-guided-project-docents-golden-dataset/The capstone for Module 2. You define a pydantic TestCase schema, author 14 validated Docent cases spanning all eight Meridian doc pages — with paraphrase pairs, an out-of-scope refusal, and an adversarial false-premise case — audit coverage and de-duplicate, then serialize the set to JSONL in a temp directory and read it back with an identical round-trip. This deterministic, reusable artifact is the dataset every metric in the rest of the course scores against.