Title here
Summary here
Module · 5 lessons
LLM-as-Judge
When there's no reference answer and no formula, use a model to grade. Learn rubric and pairwise judging, and — crucially — how to calibrate the judge against humans so you can trust its scores, then build a calibrated judge for Docent.
At a glance
Level
Intermediate
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to LLM-as-Judge, the fourth module. The deterministic metrics from Module 3 run out exactly where quality gets interesting: open-ended answers, legitimate paraphrases, and anything where “correct” depends on meaning rather than string overlap. When there’s no clean reference and no formula, you reach for a different instrument — a language model asked to grade the output. Used carelessly it produces confident nonsense; used well it’s one of the most powerful tools in evaluation.
You’ll learn when an LLM judge is the right call and when it isn’t, how to write a rubric and get structured direct scores with rationales, and how to run pairwise comparisons to pick the better of two answers. Then comes the part most teams skip and most need: the judge is itself a model that can be biased — it favors longer answers, the first option it sees, its own style — so you’ll learn to calibrate it against human labels, measure judge–human agreement, and reduce those biases. The guided project builds a calibrated judge for Docent that you actually have reason to trust.
Start with Lesson 1, where you’ll hand a hard-to-score answer to a model and watch it do what exact match couldn’t.
Lessons in this module
1
Why and When to Use an LLM Judge
Understand when a model should grade a Docent answer that a formula cannot, build a live reference-based judge with claude-haiku-4-5, and watch it fix the exact cases where token F1 was misled.
2
Rubric & Direct Scoring
Turn a noisy 'rate this 1-5' into reliable, explainable scores: build a three-criterion rubric for Docent answers and a judge that returns structured per-criterion JSON with a rationale.
3
Pairwise Comparison & Preference
Judge Docent by asking a model which of two answers is better, aggregate into a win-rate, and defeat position bias by running both orders on real claude-haiku-4-5 calls.
4
Calibrating & Trusting the Judge
Treat the judge as a classifier, score it against a small set of human gold labels, and measure agreement and Cohen's kappa so you know whether to trust its verdicts.
5
Guided Project: A Calibrated Judge for Docent
Build a trustworthy LLM judge for Docent end to end: a rubric-scoring judge that reasons before it votes, run over the golden set, then calibrated against human labels with agreement and Cohen's kappa before you ever report a score.
Achievement
Complete all 5 lessons to finish the LLM-as-Judge module.
