Title here
Summary here
Module · 5 lessons
Deterministic & Reference Metrics
When you have a reference answer, you can measure quality with code instead of judgement. Learn exact match and normalization, token-level F1, ROUGE and BLEU, and structured-output checks — then build a reference-metric scorecard for Docent.
At a glance
Level
Intermediate
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Deterministic & Reference Metrics, the third module. When a test case comes with a reference answer, you don’t need a model or a human to score it — you can measure quality with plain code that runs in milliseconds and gives the same number every time. These metrics are the fast, cheap, reproducible backbone of any eval suite, and knowing their limits is as important as knowing how to compute them.
You’ll start with exact match and the normalization that makes it fair, add token-level F1 so a mostly-right answer earns partial credit, then meet ROUGE and BLEU for longer generated text — and see clearly where each metric is trustworthy and where it quietly misleads for question answering. You’ll also learn to check structured outputs against a schema, so a Docent answer that’s supposed to be JSON with an answer and a doc_id actually is. The module ends with a guided project that assembles these into a real reference-metric scorecard for Docent.
Start with Lesson 1, where a single correct answer and a strict string comparison turn out to disagree more than you’d expect.
Lessons in this module
1
Exact Match & Normalization
Learn exact match, the simplest reference metric, and the SQuAD-style normalization that keeps it from failing correct Docent answers over trivial formatting.
2
Token-Level F1 & Overlap Metrics
Learn token-level F1, the SQuAD partial-credit metric, and score real Docent answers where exact match gives a flat zero to a mostly-correct reply.
3
ROUGE, BLEU & N-gram Metrics
Learn how the classic n-gram overlap metrics ROUGE and BLEU work, compute them on real Docent answers with the rouge-score and nltk libraries, and see exactly where they mislead for short factual QA.
4
Structured-Output & Schema Checks
Evaluate LLM outputs that must follow a structure, not free text: build a pydantic validator for Docent's JSON answers and report a structured-output pass rate over live and hand-crafted cases.
5
Guided Project: A Reference-Metric Scorecard
Build a real scorecard that runs Docent over the golden set and reports exact match, token F1, and ROUGE-L side by side, then read the disagreement between them.
Achievement
Complete all 5 lessons to finish the Deterministic & Reference Metrics module.
