Title here
Summary here
Course
LLM Evaluation & Observability
Stop guessing whether your LLM app works — build a real evaluation and observability practice with datasets, metrics, LLM-as-judge, RAG and agent evals, tracing, and production monitoring, hands-on in Python with Claude
At a glance
Level
Intermediate
Lessons
41 lessons across 9 modules
What you build
An eval & observability harness for an LLM app
Cost
Free · live lessons use low-cost
claude-haiku-4-5What you'll build
You'll evaluate and monitor Docent, a documentation Q&A assistant built on Claude that answers questions from a product's docs. Instead of eyeballing a few answers and calling it "good enough," you'll build a real evaluation and observability practice around it: assembling golden datasets, measuring answers with deterministic metrics (exact match, F1, ROUGE) and structured-output checks, scoring open-ended answers with a calibrated LLM-as-judge, evaluating RAG for faithfulness and retrieval quality, checking agent task success and tool-call correctness, then wiring up tracing, cost/latency/token accounting, production monitoring, guardrails, and CI regression tests. The deterministic metric code runs for real with rouge-score and scikit-learn; the model-graded parts call Claude live on the low-cost claude-haiku-4-5 model, so the numbers you see are real measurements.
Course syllabus
Work through the modules at your own pace. Each lesson is a self-contained, hands-on read.
1 Why Evaluation Matters
2 Building Eval Datasets
3 Deterministic & Reference Metrics
4 LLM-as-Judge
5 Evaluating RAG
6 Evaluating Agents & Tool Use
7 Observability & Tracing
8 Production Monitoring & Guardrails
Before you start
You'll need comfortable Python and a working understanding of building LLM applications — prompting, calling a model from code, retrieval-augmented generation, and tool use. Our Generative AI & LLM Engineering and Building AI Agents in Python courses are ideal preparation. Those courses teach you to build LLM apps and agents; this one teaches you to measure, trust, and monitor them — the discipline that separates a demo from something you'd run in production.
Set up your environment
You can complete this course on any machine with Python 3.10+. The deterministic metric lessons need no API key at all. The model-graded lessons (LLM-as-judge, RAG and agent evaluation) call Claude, so you'll want an Anthropic API key for those — the course uses the low-cost claude-haiku-4-5 model throughout, so running every live example end to end costs only a few cents.
- Install the packages the course uses:
pip install anthropic rouge-score scikit-learn nltk pydantic numpy- For the live model lessons, set your key in the environment (never hard-code it in a script):
export ANTHROPIC_API_KEY="your-key-here"Model outputs are non-deterministic, so your model-graded numbers will land close to — not identical to — the ones shown. The deterministic metric outputs will match exactly. API surfaces shift over time; if a signature has changed since these versions, the evaluation concepts still apply — adjust the syntax to what you have installed.
Ready to stop guessing whether your LLM app works?
Start with the "looks fine" trap — why reading a few answers fools everyone — and build a real evaluation and observability practice from datasets and metrics all the way to production monitoring.
Start the first lessonWant this taught live to your team?
Mehdi runs tailored corporate workshops on this exact material — hands-on, in-person or remote.
