https://datatweets.com/courses/llm-evaluation/deterministic-and-reference-metrics/Recent content in Deterministic & Reference Metrics on DATATWEETSHugoenCopyright (c) 2026 DatatweetsTue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/deterministic-and-reference-metrics/lesson-1-exact-match-and-normalization/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/deterministic-and-reference-metrics/lesson-1-exact-match-and-normalization/Exact match is the cheapest, most reproducible metric there is – output equals reference, scored by code in microseconds – but raw string equality is brutally strict: ‘HTTP 429.’ and ‘429’ are the same fact and it calls them different. This lesson implements the classic SQuAD normalize_answer (lowercase, strip punctuation, drop articles, collapse whitespace) and exact_match, scores a realistic table of Docent predictions against the canonical golden references, and watches the aggregate jump from a raw 2/8 = 0.250 to a normalized 7/8 = 0.875. It also shows exact match’s hard limit – no partial credit – which motivates token-level F1 next.https://datatweets.com/courses/llm-evaluation/deterministic-and-reference-metrics/lesson-2-token-level-f1/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/deterministic-and-reference-metrics/lesson-2-token-level-f1/Exact match is all-or-nothing, so a mostly-right Docent answer scores the same zero as a completely wrong one. Token-level F1 fixes that by treating prediction and reference as bags of normalized tokens and computing precision, recall, and their harmonic mean. This lesson derives the SQuAD F1 formula, builds a real deterministic f1_score function, and scores four Docent answers – an exact hit, a verbose-but-correct reply, a terse partial answer, and a wrong one – to show mean F1 (0.51) rewarding partial credit where mean EM (0.25) cannot. It also exposes F1’s blind spots: bag-of-words ignores word order and meaning, which motivates the semantic and judge metrics later in the course.https://datatweets.com/courses/llm-evaluation/deterministic-and-reference-metrics/lesson-3-rouge-bleu-and-ngram-metrics/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/deterministic-and-reference-metrics/lesson-3-rouge-bleu-and-ngram-metrics/ROUGE and BLEU score generated text by counting overlapping n-grams against a reference, and they are the backbone of summarization and translation evaluation. This lesson builds the n-gram intuition by hand, then computes real ROUGE-1/2/L and BLEU on three Docent answers to the same Meridian question – a correct answer, a correct paraphrase, and a fluent-but-wrong answer. The wrong answer scores highest on both metrics because it reuses the reference’s words while flipping the meaning, which is the concrete lesson: surface overlap is not correctness, and these metrics fit longer summaries far better than short factoid QA.https://datatweets.com/courses/llm-evaluation/deterministic-and-reference-metrics/lesson-4-structured-output-and-schema-checks/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/deterministic-and-reference-metrics/lesson-4-structured-output-and-schema-checks/When an LLM feature must emit machine-readable output, correctness starts with ‘parses and validates.’ This lesson makes Docent answer as JSON with an answer, a doc_id constrained to Meridian’s 8 pages, and a confidence in [0,1], then grades it with a five-rung check ladder: does it parse, are the required fields present, are the types right, are enums and ranges valid, do cross-field constraints hold. You’ll build a pydantic validator with a robust extract_json() that strips markdown fences, run it live on four questions for a real pass rate, and run a reproducible deterministic block over good and broken JSON strings.https://datatweets.com/courses/llm-evaluation/deterministic-and-reference-metrics/lesson-5-guided-project-reference-metric-scorecard/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/deterministic-and-reference-metrics/lesson-5-guided-project-reference-metric-scorecard/The capstone for Module 3. You assemble the module’s three deterministic metrics — normalized exact match, token-level F1, and ROUGE-L via the rouge-score library — behind SQuAD-style normalization, run Docent live over an eight-question golden set, then score every answer on every metric to build a per-item table and aggregate scorecard. Finally you read it: why EM, F1, and ROUGE-L disagree, which to trust for short factual answers, and where all three fall short — motivating LLM-as-judge in Module 4.