https://datatweets.com/courses/llm-evaluation/evaluating-rag/Recent content in Evaluating RAG on DATATWEETSHugoenCopyright (c) 2026 DatatweetsTue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/evaluating-rag/lesson-1-the-rag-eval-problem/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/evaluating-rag/lesson-1-the-rag-eval-problem/Docent is a retriever plus a generator wired in series, so a wrong answer has two possible causes and reading the final answer alone can’t tell them apart. You’ll decompose Docent into its retrieve() step and its Claude generation step, map every failure into four quadrants, and reproduce the Module 1 bug live: the keyword retriever pulls [’errors’,‘backups’] for ‘What does Meridian charge for the Free plan?’ — never the plans page — producing a false refusal that looks like a generation flaw but is pure retrieval. That is the diagnosis this whole module teaches you to make.https://datatweets.com/courses/llm-evaluation/evaluating-rag/lesson-2-retrieval-metrics/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/evaluating-rag/lesson-2-retrieval-metrics/A RAG answer can only be as good as the documents the retriever hands the model, so before you judge Docent’s answers you measure its search. This lesson defines and implements the four canonical retrieval metrics – Recall@k (did a relevant doc come back at all), Precision@k (how much of the top-k was relevant), MRR (how high was the first relevant doc ranked), and NDCG@k (rank quality against the ideal ordering) – and runs them deterministically over a 9-question golden set where each question maps to one correct Meridian doc. The ‘Free plan’ question exposes the keyword-retriever bug from Module 1: its relevant page lands at rank 5, dragging Recall@2 from a perfect 1.000 down to 0.889 while MRR (0.800) and NDCG@2 (0.807) still register a mostly-good ranking. You’ll see precision, recall, and MRR tell three different stories about the same retriever and learn which one to trust for which question.https://datatweets.com/courses/llm-evaluation/evaluating-rag/lesson-3-faithfulness-and-groundedness/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/evaluating-rag/lesson-3-faithfulness-and-groundedness/A RAG answer can read perfectly and still smuggle in a fact the retrieved context never stated. This lesson builds a live faithfulness checker for Docent: prompt claude-haiku-4-5 to split an answer into atomic claims and label each SUPPORTED or NOT_SUPPORTED against the retrieved Meridian page, then compute faithfulness as supported over total. You’ll watch a grounded answer score 1.00 while an answer that invents a free trial and an uptime SLA drops to 0.33 with the fabricated claims flagged, and you’ll separate faithfulness from correctness with a concrete example: an answer can be faithful but wrong, or correct but unfaithful.https://datatweets.com/courses/llm-evaluation/evaluating-rag/lesson-4-answer-relevance-and-context-quality/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/evaluating-rag/lesson-4-answer-relevance-and-context-quality/Retrieval metrics and faithfulness only cover half of RAG quality. This lesson adds the two signals that finish the RAGAS quartet: answer relevance – does the answer respond to what was asked, or is it evasive and off-topic even when it is faithful – and context precision plus context recall, which measure whether retrieval pulled useful chunks and whether it missed a page the answer needed. You’ll build a live LLM judge that scores answer relevance for on-topic, evasive, and off-topic Docent answers, compute context precision and recall deterministically from retrieved doc ids and the golden set (including the Free-plan question where recall is zero because retrieval missed the plans page), and assemble a per-question RAG report whose four signals localize each failure to the retriever or the generator.https://datatweets.com/courses/llm-evaluation/evaluating-rag/lesson-5-guided-project-full-rag-evaluation/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/evaluating-rag/lesson-5-guided-project-full-rag-evaluation/The capstone for Module 5. You assemble the module into one runnable harness that evaluates Docent’s retriever and generator together, then pinpoints where each failure comes from. You build one evaluation record per golden question, compute recall@k, precision@k, and MRR deterministically, score faithfulness and answer relevance with a live judge on claude-haiku-4-5, and finally build a per-question diagnosis table that classifies each failure as RETRIEVAL (the relevant doc never came back) or GENERATION (the context was right, the answer wasn’t). The payoff is diagnosing Docent’s Free-plan false refusal as a retrieval bug you fix in the search, not the prompt.