https://datatweets.com/courses/llm-evaluation/why-evaluation-matters/Recent content in Why Evaluation Matters on DATATWEETSHugoenCopyright (c) 2026 DatatweetsTue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/why-evaluation-matters/lesson-1-the-looks-fine-trap/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/why-evaluation-matters/lesson-1-the-looks-fine-trap/You’ll run Docent, our Meridian documentation assistant, on questions it clearly nails and declare it working. Then you’ll push a few more golden questions through and catch it giving a confident-sounding but wrong refusal that spot-checking two answers would have missed. A little probability math shows why: even a 70%-correct assistant looks perfect in three checks about a third of the time. The fix, previewed here and built across the course, is datasets and metrics instead of vibes.https://datatweets.com/courses/llm-evaluation/why-evaluation-matters/lesson-2-offline-versus-online-evaluation/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/why-evaluation-matters/lesson-2-offline-versus-online-evaluation/This lesson introduces offline evaluation (running Docent over a fixed golden set before you ship, for reproducible, cheap comparisons that gate a release) and online evaluation (measuring Docent on real production traffic after you ship, using implicit signals like thumbs, edits, and refusal rates). You will run a real offline pass over four golden questions and see 4 of 4 pass, then watch an illustrative live stream where a thumbs-down exposes a case the golden set never contained. It closes with the core trade-offs: coverage versus realism, speed versus signal, and reproducibility versus relevance.https://datatweets.com/courses/llm-evaluation/why-evaluation-matters/lesson-3-the-evaluation-mindset/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/why-evaluation-matters/lesson-3-the-evaluation-mindset/Move from ’the answers look fine’ to a number you can defend. This lesson names the four components every evaluation needs — a task, a dataset, a metric, and a target with an aggregation rule — and walks the eval loop that ties them together. You then build a real, generic run_eval harness and run it on a tiny in-memory Docent dataset with a deterministic substring metric, so it reproduces exactly and needs no API key. You’ll also see absolute versus regression evals and the beginner mistakes that quietly break both.https://datatweets.com/courses/llm-evaluation/why-evaluation-matters/lesson-4-what-to-measure-quality-dimensions/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/why-evaluation-matters/lesson-4-what-to-measure-quality-dimensions/‘Good’ is not one number. This lesson splits the quality of Docent, our Meridian docs assistant, into six dimensions — correctness, faithfulness, relevance, format, safety, and operational cost/latency — each needing a different method and a different later module. You will map every dimension to a concrete Docent example, run a real live call that measures the operational dimension (latency and token counts), and see why a single blended average hides the trade-offs that matter.https://datatweets.com/courses/llm-evaluation/why-evaluation-matters/lesson-5-guided-project-your-first-eval-harness/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/why-evaluation-matters/lesson-5-guided-project-your-first-eval-harness/The capstone for Module 1. You assemble an 8-item golden dataset (including an out-of-scope refusal probe), run Docent over it with claude-haiku-4-5, score each answer with a deliberately crude keyword-and-refusal metric, and aggregate the results into a per-item table and a single pass rate. This four-stage harness is the exact scaffold every later module extends.