https://datatweets.com/courses/llm-evaluation/llm-as-judge/Recent content in LLM-as-Judge on DATATWEETSHugoenCopyright (c) 2026 DatatweetsTue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/llm-as-judge/lesson-1-why-and-when-to-use-an-llm-judge/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/llm-as-judge/lesson-1-why-and-when-to-use-an-llm-judge/Deterministic metrics from Module 3 cannot tell a correct paraphrase from a wrong answer – a fluent answer that flips the meaning can still out-score a right one on F1. This lesson introduces the LLM-as-judge pattern: prompt a capable model with the question, the answer, and a reference, and ask for a correct/incorrect verdict with a reason. You build a live reference-based judge with claude-haiku-4-5 on four Docent items where F1 fails, and see the judge grade a correct paraphrase (F1 0.15) as correct and a fluent wrong answer (F1 0.80) as incorrect. It also covers the three judging modes the module explores, the judge’s real risks, and when a deterministic check is the better, cheaper tool.https://datatweets.com/courses/llm-evaluation/llm-as-judge/lesson-2-rubric-and-direct-scoring/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/llm-as-judge/lesson-2-rubric-and-direct-scoring/A bare ‘rate this answer 1-5’ drifts run to run and tells you nothing about what was wrong. This lesson builds a rubric that defines each criterion — Correctness, Groundedness, Helpfulness — and what every score level means, so a judge maps an answer to an explicit rung instead of guessing a number. You’ll write judge_answer() calling claude-haiku-4-5 for a structured JSON verdict with the rationale before the scores, validate and parse that JSON, aggregate per criterion, and run it live on four Docent answers — a strong one, a vague one, a hallucinated one, a wrong-fact one — watching groundedness collapse on the invented answer while correctness stays higher elsewhere.https://datatweets.com/courses/llm-evaluation/llm-as-judge/lesson-3-pairwise-comparison-and-preference/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/llm-as-judge/lesson-3-pairwise-comparison-and-preference/Instead of scoring one answer in isolation, pairwise judging asks the model which of two answers is better and aggregates the verdicts into a win-rate – the backbone of preference evaluation and leaderboards. This lesson builds a real judge_pairwise on claude-haiku-4-5, runs a good Docent variant against a vague one to get a 0.75 win-rate, then exposes the headline pitfall: on near-equal answers the judge flips its verdict one to two times out of four just by swapping the order, and one pair flips on every single run. You fix it by running both orders and counting only order-consistent verdicts, and meet the other biases – verbosity, ties, and broken transitivity.https://datatweets.com/courses/llm-evaluation/llm-as-judge/lesson-4-calibrating-and-trusting-the-judge/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/llm-as-judge/lesson-4-calibrating-and-trusting-the-judge/A judge is a model, and a model can be confidently, systematically wrong. This lesson gives you the discipline that makes an LLM judge usable: calibrate it against human labels. You’ll meet the biases that break naive judges — position, verbosity, self-preference, leniency and severity, prompt sensitivity — then treat the judge as a classifier and score it against a ten-item human gold set for Docent. You’ll compute raw agreement and Cohen’s kappa for a naive judge (0.40 agreement, kappa minus 0.07 — worse than chance) and a grounded judge (1.00 agreement, kappa 1.00), watch the fix move every item onto the diagonal, and run a verbosity-bias probe.https://datatweets.com/courses/llm-evaluation/llm-as-judge/lesson-5-guided-project-calibrated-judge/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/llm-as-judge/lesson-5-guided-project-calibrated-judge/The capstone for Module 4. You assemble the module into one runnable artifact: a judge_answer function on claude-haiku-4-5 that returns validated JSON with reasoning before a correct/incorrect verdict, run live over Docent’s ten-item golden set, then calibrated against hand-authored human gold labels on a mixed set of right and wrong answers. You compute judge-human agreement and Cohen’s kappa, set a trust threshold, inspect a real disagreement, and finally report Docent’s judged quality score with its calibration caveat attached – never a judge score without the number that says how far to trust it.