Title here
Summary here
Module · 5 lessons
Building Eval Datasets
An evaluation is only as good as the data behind it. Learn what makes a trustworthy golden set, how to write test cases by hand, how to mine production logs into test sets, and how to keep a dataset covered, sliced, and clean — then build Docent's golden dataset.
At a glance
Level
Intermediate
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Building Eval Datasets, the second module. Every metric you’ll learn in this course scores an output against an example, which means your evaluation is only ever as good as the dataset behind it. A biased, thin, or leaky dataset produces confident numbers that mean nothing. This module makes your dataset something you can trust.
You’ll learn what separates a real golden set from a handful of cherry-picked questions — representativeness, labeled references, diversity, and hard cases held out on purpose. You’ll write test cases by hand with structured references and tags, then learn to mine production logs into fresh test cases so your dataset tracks real usage. You’ll measure coverage and slices to find the blind spots a single average hides, and practice the hygiene — versioning, deduplication, no leakage — that keeps a dataset honest over time. The guided project assembles Docent’s golden dataset: versioned, tagged, covered, and ready to drive every metric in the modules that follow.
Start with Lesson 1, where you’ll learn what actually makes an evaluation dataset trustworthy.
Lessons in this module
1
Golden Sets: What Makes a Good Eval Dataset
Learn the five properties of a trustworthy golden set for evaluating Docent, and run a real dataset-health audit over the canonical Meridian Q&A set.
2
Writing Test Cases by Hand
Author high-quality Docent test cases from scratch — the seven fields of a case, the five case types to include on purpose, checkable references, and a pydantic schema that rejects bad cases.
3
From Production Logs to Test Sets
Turn Docent's real Meridian production traffic into a test set: capture structured logs, deduplicate near-identical questions, sample failures, label references, and promote past failures into regression cases.
4
Coverage, Slicing & Dataset Hygiene
Measure whether Docent's golden set actually covers every Meridian doc page, report accuracy per slice to expose a weak 'hard' subgroup, and keep the dataset clean with dedup and leakage checks.
5
Guided Project: Build Docent's Golden Dataset
Assemble Docent's real, versioned golden dataset: a pydantic-validated schema, 14 authored cases across all 8 Meridian doc pages, a coverage audit, and a JSONL round-trip you can reuse in every later module.
Achievement
Complete all 5 lessons to finish the Building Eval Datasets module.
