https://datatweets.com/courses/llm-evaluation/evaluating-agents-and-tool-use/Recent content in Evaluating Agents & Tool Use on DATATWEETSHugoenCopyright (c) 2026 DatatweetsTue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/evaluating-agents-and-tool-use/lesson-1-why-agents-are-harder-to-evaluate/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/evaluating-agents-and-tool-use/lesson-1-why-agents-are-harder-to-evaluate/A single-pass assistant produces one output you can grade with one check. An agent produces a trajectory: a sequence of tool calls and observations before the final answer, and every step is a new place to fail. You’ll turn Docent into a tool-using agent that decides for itself when to call search_docs, run it live on three questions, and print the recorded trajectory of each. You’ll watch a clean single-search-then-answer path, and a wandering out-of-scope path that searches four or five times before refusing, then see how many of those failure points a final-answer check would never catch, motivating the module’s split into outcome and process evaluation.https://datatweets.com/courses/llm-evaluation/evaluating-agents-and-tool-use/lesson-2-task-success-and-outcome-metrics/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/evaluating-agents-and-tool-use/lesson-2-task-success-and-outcome-metrics/The single most important number for an agent is its task success rate: the fraction of tasks it actually accomplishes, judged by a checkable per-task criterion. This lesson turns the agentic Docent loose on eight golden tasks (seven factual questions plus one out-of-scope probe that should be refused), defines a task_succeeded verifier that checks the final answer against a reference fact (or checks for a correct refusal), and computes the success rate with a real per-task pass/fail board – 8/8 on a live run. To prove the metric earns its trust, we inject a deliberately broken retriever into the same agent and re-run: the answers stop containing the reference facts and success collapses to 1/8, which the outcome metric flags immediately. Along the way you’ll separate outcome from process, choose between exact-state checks, verifier functions, and LLM judges, and weigh binary success against partial credit for multi-step tasks.https://datatweets.com/courses/llm-evaluation/evaluating-agents-and-tool-use/lesson-3-tool-call-correctness/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/evaluating-agents-and-tool-use/lesson-3-tool-call-correctness/A tool-using agent can land the right answer through a wrong tool call, so this lesson scores the tool calls on their own. You run agentic Docent live on a handful of Meridian questions, record each trajectory’s search_docs calls, then apply three deterministic gates: called_search (did it search at all), args_valid (is the query a non-empty string per the tool schema), and a query-quality heuristic (does the query share a keyword with the question). Over a fixed list of 9 recorded tasks the checkers report tool-call accuracy 0.778, tool-selection precision and recall both 0.857, an argument-validity rate of 0.857, and a query-quality rate of 0.571 – byte-identical across two runs. Two hand-built negative trajectories show the failures that matter most: answering from parametric memory with no search call at all (a costly false negative) and calling the tool with an empty, useless query.https://datatweets.com/courses/llm-evaluation/evaluating-agents-and-tool-use/lesson-4-trajectory-evaluation/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/evaluating-agents-and-tool-use/lesson-4-trajectory-evaluation/Two agents can both reach the right answer, but one takes a single clean step and the other wanders through a redundant loop before recovering. Outcome checks and per-call checks never see that difference; the trajectory does. This lesson builds deterministic trajectory metrics over Docent’s recorded steps – num_tool_calls for efficiency, has_redundant_call to catch loops, and matches_reference against the expected ‘one search then answer’ path – scoring a clean live trajectory against a hand-built wasteful one. Then it adds a live claude-haiku-4-5 judge that rates the process 1-5 with a reason, giving the clean path a 5 and the wasteful path a 3, honest about run-to-run wobble.https://datatweets.com/courses/llm-evaluation/evaluating-agents-and-tool-use/lesson-5-guided-project-evaluating-an-agentic-docent/Tue, 07 Jul 2026 09:00:00 +0200https://datatweets.com/courses/llm-evaluation/evaluating-agents-and-tool-use/lesson-5-guided-project-evaluating-an-agentic-docent/The capstone for Module 6. You take the agentic Docent — the version that decides for itself to call search_docs — and evaluate it end to end at every level the module taught. You build a task set of eight golden questions (seven factual plus the out-of-scope refusal), run the agent live on each while recording its answer and full trajectory, then score all three levels deterministically over the collected runs: task success from the final answer, tool-call correctness from the arguments, and trajectory quality from the path. The payoff is an agent scorecard that shows every task passing on outcome and tool calls while the trajectory level exposes the real weakness — an agent that reaches the right answer down a wasteful path.