Title here
Summary here
Module · 5 lessons
Why Evaluation Matters
The case for evaluating LLM apps: why reading a few answers fools everyone, offline versus online evaluation, the eval mindset, and what quality dimensions to measure — then build your first end-to-end eval harness for Docent.
At a glance
Level
Intermediate
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Why Evaluation Matters, the opening module of the course. Before you can measure an LLM app well, you have to understand why measuring it at all is harder — and more necessary — than it looks. This module makes that case and gives you the vocabulary and mindset the rest of the course builds on.
You’ll start with the “looks fine” trap: read a few of Docent’s answers, decide it works, and then watch it fail on a question you didn’t happen to check. From there you’ll learn the difference between offline evaluation (on datasets, before you ship) and online evaluation (on real traffic, after you ship), the mindset that replaces gut feel with repeatable metrics, and the quality dimensions — correctness, faithfulness, relevance, format, cost, latency — that a serious eval has to cover. The module closes with a guided project where you build a small but real evaluation harness for Docent, end to end.
Start with Lesson 1, and let Docent fool you before you learn not to be fooled.
Lessons in this module
1
The "Looks Fine" Trap
Watch Docent answer a few Meridian questions correctly, then slip on one you didn't check, and see why eyeballing a handful of outputs gives false confidence.
2
Offline vs. Online Evaluation
Learn the two complementary regimes of LLM evaluation by scoring Docent on a fixed golden set before shipping and measuring it on real user traffic after.
3
The Evaluation Mindset
Turn subjective judgement of Docent's answers into a repeatable measurement by learning the four components of an eval and building a real, key-free eval harness in Python.
4
What to Measure: Quality Dimensions
Break 'is it good?' into six measurable quality dimensions for Docent — correctness, faithfulness, relevance, format, safety, and operational cost — and see which module measures each.
5
Guided Project: Your First Eval Harness
Build a small but real end-to-end evaluation harness for Docent: a golden dataset, a run loop over Claude, an honest keyword scorer, and an aggregate pass-rate report.
Achievement
Complete all 5 lessons to finish the Why Evaluation Matters module.
