Title here
Summary here
Module · 5 lessons
Evaluating Agents & Tool Use
An agent doesn't just produce an answer — it takes a path of tool calls to get there, and that path can fail in ways a final-answer check never sees. Learn task success, tool-call correctness, and trajectory evaluation, then evaluate an agentic Docent end to end.
At a glance
Level
Intermediate
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Evaluating Agents & Tool Use, the sixth module. So far Docent has answered in a single pass. But a real assistant is often an agent: it decides for itself to search the docs, reads what comes back, maybe searches again, and only then answers. That autonomy adds an entire new surface to evaluate — the path the agent takes — and a wrong answer can now hide a wrong tool, wrong arguments, a redundant loop, or a premature give-up that a final-answer check would never reveal.
You’ll learn to evaluate agents at three levels: task success (did it actually accomplish the goal?), tool-call correctness (did it call the right tool with valid arguments?), and trajectory evaluation (was the sequence of steps efficient and sound, or did it wander?). Each answers a different question, and you need all three: an agent can reach the right answer by a terrible path, or take a perfect path to the wrong place. The guided project turns Docent into a tool-using agent and evaluates it end to end — success, tool calls, and trajectories together.
Start with Lesson 1, where a single answer becomes a whole path — and every step of it can go wrong.
Lessons in this module
1
Why Agents Are Harder to Evaluate
See why grading an agent means grading a whole trajectory of tool calls, not one answer, by running a tool-using Docent live and printing the path it takes to each answer.
2
Task Success & Outcome Metrics
Measure the headline agent metric -- task success rate -- by running the agentic Docent over a golden task set, defining a per-task verifier, and watching a broken-retriever regression drop success from 8/8 to 1/8.
3
Tool-Call Correctness
Evaluate an agent's tool calls independently of its final answer -- did Docent call search_docs, with a schema-valid and genuinely useful query -- using deterministic checkers over recorded trajectories.
4
Trajectory Evaluation
Evaluate the whole path an agent takes, not just its answer: count tool calls, detect redundant loops, match against an expected trajectory, and score process quality with an LLM judge.
5
Guided Project: Evaluating an Agentic Docent
Evaluate the tool-using agentic Docent end to end across all three module levels — task success, tool-call correctness, and trajectory quality — with live claude-haiku-4-5 runs and deterministic scoring over the collected trajectories.
Achievement
Complete all 5 lessons to finish the Evaluating Agents & Tool Use module.
