Title here
Summary here
Module · 5 lessons
Evaluating RAG
A RAG system is two systems in one — a retriever and a generator — and each fails differently. Learn retrieval metrics, faithfulness and groundedness, answer relevance and context quality, then run a full RAG evaluation on Docent that pinpoints where failures come from.
At a glance
Level
Intermediate
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Evaluating RAG, the fifth module. Docent is a retrieval-augmented system: it retrieves the Meridian doc pages most relevant to a question, then generates an answer from them. That means a wrong answer has two possible causes — the retriever fed the model the wrong context, or the model mishandled the right context — and if you only measure the final answer, you can’t tell which. This module teaches you to evaluate both halves.
You’ll learn retrieval metrics — precision, recall, MRR, and NDCG — to score whether the right documents came back, then faithfulness and groundedness to catch a generator that invents claims the context doesn’t support, and finally answer relevance and context quality to judge the pipeline as a whole. Along the way you’ll reconnect with a bug you met in Module 1: Docent’s keyword retriever quietly pulls the wrong pages for certain questions. The guided project runs a full RAG evaluation on Docent and, crucially, diagnoses each failure as retrieval or generation — the difference between fixing your search and fixing your prompt.
Start with Lesson 1, where one system becomes two — and each gets its own scorecard.
Lessons in this module
1
The RAG Eval Problem: Two Systems in One
See why a RAG app is really two systems in series, reproduce Docent's deterministic retrieval bug on the Free-plan question, and learn to localize a bad answer to retrieval or generation.
2
Retrieval Metrics: Precision, Recall, MRR, NDCG
Score Docent's retriever with the four classic information-retrieval metrics -- Recall@k, Precision@k, MRR, and NDCG@k -- built for real over a golden set and quantifying the Module-1 keyword-retrieval bug.
3
Faithfulness & Groundedness
Measure whether Docent's answer is faithful to the retrieved Meridian context: decompose it into atomic claims and check that each one is entailed by the context, with nothing invented.
4
Answer Relevance & Context Quality
Complete the RAG metric picture: score whether Docent's answer actually addresses the question (answer relevance) and whether retrieval fetched good, complete context (context precision and recall).
5
Guided Project: A Full RAG Evaluation
Evaluate Docent's RAG pipeline end to end and diagnose every failure as a retrieval problem or a generation problem, combining deterministic retrieval metrics with a live claude-haiku-4-5 judge for faithfulness and answer relevance.
Achievement
Complete all 5 lessons to finish the Evaluating RAG module.
