Lesson 1 - Why Agents Are Harder to Evaluate

Welcome to Why Agents Are Harder to Evaluate#

For five modules you have graded Docent — our Meridian documentation assistant — by looking at the answer it produced. Retrieval happened for it: some code fetched two doc pages, handed them to Claude, and Claude wrote a reply. There was exactly one model output per question, and evaluation meant asking one question of it: is that answer right?

This module changes the shape of the thing you are grading. A real assistant is often an agent: instead of being handed context, it decides for itself to search the docs, reads what comes back, maybe searches again, and only then answers. That autonomy is powerful, but it means the agent no longer produces a single output. It produces a trajectory — a sequence of tool calls and observations that leads to the final answer. And a trajectory can go wrong in far more places than a single answer can. This lesson makes that concrete by turning Docent into a tool-using agent, running it live, and printing the path it takes so you can see every step that could have failed.

In this lesson, you will:

• See why an agent’s output is a trajectory of tool calls and observations, not one answer
• Enumerate the ways a trajectory fails: wrong tool, wrong arguments, wrong order, redundant loops, giving up early, hitting a step limit
• Understand the difference between outcome (was the answer right?) and process (was the path sound?), and why you need both
• Run the agentic Docent live and print its recorded trajectory on a clean question and a wandering out-of-scope one
• Preview the module: task success, tool-call correctness, and trajectory evaluation

One answer versus a whole path#

Start with what you have been grading. A single-pass system takes a question, makes one model call, and emits one answer. To evaluate it you compare that answer to a reference or judge it on its own. There is exactly one artifact and one place to look.

An agent replaces that single call with a loop. Given a question, the model can choose to call a tool — here, search_docs — instead of answering. Your code runs the tool, feeds the result back, and asks the model again. It might search once and answer, search twice with a refined query, or keep going until it either answers or hits a hard cap on steps. What you get back is not a string; it is a record of everything the agent did: which tools it called, with which arguments, what came back, and how it finally decided to respond.

That record is the trajectory, and it is why agents are harder to evaluate. Count the ways a single answer can be wrong: essentially one — it is right or it is not. Now count the ways a trajectory can be wrong:

• Wrong tool chosen — it answered from memory without searching, or reached for a tool the task didn’t need.
• Right tool, wrong arguments — it searched, but with a query so off that the useful page never came back.
• Right steps, wrong order — it answered before it had the observation it needed, or searched after it had already decided.
• Redundant loops — it searched the same idea three times, burning calls and latency for nothing.
• Gave up too early — it refused or guessed when one more search would have found the answer.
• Hit the step limit — it looped until the cap and returned no real answer at all.

Every one of those is a real failure, and — this is the crucial part — most of them are invisible if you only read the final answer. An agent that searches four times in a redundant loop and then answers correctly looks, from the outside, identical to one that searched once and answered correctly. Same answer, very different behavior. Grade only the answer and you will ship the wasteful, brittle agent and never know.

Outcome versus process#

The figure names the two things you now have to evaluate, and it is worth stating them plainly because the whole module hangs on the distinction.

Outcome is the classic question: did the agent achieve the task? For Docent, did it end up with the right answer, or correctly refuse when the docs can’t help? This is what you have been measuring all along.

Process is the new question: was the path to that outcome sound? Did it call the right tool, with sensible arguments, an appropriate number of times, in an order that makes sense? Process is about the trajectory, not the destination.

You need both, because the two come apart in exactly the ways that cost you in production:

• Right by luck (bad path, right answer). The agent stumbles through a wrong-looking sequence — searches with a vague query, gets mediocre pages — and still, by the model’s general knowledge or a lucky overlap, produces the correct answer. Outcome says “pass.” Process says “this will not hold.” On a slightly different question the same broken path will fail, and your outcome metric gave you no warning.
• Perfect path, wrong answer. The agent does everything right — picks the tool, writes a good query, gets the correct page — and then misreads it, producing a wrong final answer. Outcome says “fail,” but the process was almost entirely correct. Knowing that tells you the fix is in the last step (reading the observation), not in the search.

An outcome-only evaluation collapses both of these into a single pass/fail on the answer and throws away the information you most need to fix the agent. That is why this module evaluates agents at multiple levels — and why the rest of it is organized around exactly this split.

Meet the agentic Docent#

To see a real trajectory we need a real agent. Here is Docent rebuilt as a tool-using agent. The retrieval logic is the same keyword-overlap search you already know, but the difference is who decides to call it: not our code, but the model. We expose search_docs as a tool, and Claude chooses when — and how many times — to use it. After each call we record a step in a trajectory list, so nothing the agent does is hidden from us.

import warnings
warnings.filterwarnings("ignore")
import anthropic, json
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

DOCS = [
    {"id": "auth", "title": "Authentication",
     "text": "Meridian authenticates every request with an API key sent in the Authorization header as a bearer token. "
             "Create keys in the dashboard under Settings > API Keys. Keys are either test or live mode; test keys only "
             "touch sandbox data. A key is shown once at creation and cannot be recovered, only rotated."},
    {"id": "ratelimits", "title": "Rate Limits",
     "text": "The Free plan allows 60 requests per minute. The Pro plan allows 600 requests per minute. Exceeding the "
             "limit returns HTTP 429 with a Retry-After header giving the seconds to wait. Rate limits are applied per "
             "API key, not per account."},
    {"id": "plans", "title": "Plans and Pricing",
     "text": "Meridian has three plans. Free costs 0 dollars per month and includes 1 GB of storage. Pro costs 25 dollars "
             "per month and includes 50 GB of storage. Scale costs 199 dollars per month and includes 500 GB of storage. "
             "All plans include unlimited read replicas."},
    {"id": "regions", "title": "Regions",
     "text": "Databases can be created in three regions: us-east, eu-west, and ap-south. The region is fixed at creation "
             "time and cannot be changed later; to move data to another region you must export and re-import into a new "
             "database. Cross-region read replicas are available on the Scale plan only."},
    {"id": "errors", "title": "Error Codes",
     "text": "Meridian uses standard HTTP status codes. 400 means a malformed request, 401 means a missing or invalid API "
             "key, 403 means the key is valid but lacks permission, 404 means the resource does not exist, and 429 means "
             "the rate limit was exceeded. 5xx codes indicate a server-side error and should be retried with backoff."},
    {"id": "backups", "title": "Backups",
     "text": "Automatic daily backups are retained for 7 days on the Free plan, 30 days on Pro, and 90 days on Scale. "
             "Backups run at 03:00 UTC. Point-in-time recovery to any second in the retention window is available on the "
             "Scale plan. Restoring a backup creates a new database and never overwrites the original."},
    {"id": "sdks", "title": "SDKs",
     "text": "Official SDKs are available for Python, JavaScript, and Go. Install the Python SDK with 'pip install meridian'. "
             "The SDK reads the MERIDIAN_API_KEY environment variable by default. All SDKs retry idempotent requests up to "
             "3 times on 5xx errors with exponential backoff."},
    {"id": "deletion", "title": "Deleting Data",
     "text": "Deleting a database is immediate and irreversible once the 24-hour grace period ends. During the grace period "
             "the database is soft-deleted and can be restored from the dashboard. After 24 hours the data and its backups "
             "are permanently purged. Deletion requires a live-mode key with admin permission."},
]

SEARCH_TOOL = [{
    "name": "search_docs",
    "description": "Search the Meridian documentation and return matching pages. Call this before answering.",
    "input_schema": {
        "type": "object",
        "properties": {"query": {"type": "string", "description": "search query"}},
        "required": ["query"],
    },
}]

def search_docs(query, docs, k=2):
    q = set(query.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())), reverse=True)
    hits = scored[:k]
    return [{"id": d["id"], "title": d["title"], "text": d["text"]} for d in hits]

def agentic_docent(question, docs, model="claude-haiku-4-5", max_steps=4):
    """Returns (final_answer, trajectory). trajectory is a list of dicts recording each step:
       {"type": "tool_call", "tool": "search_docs", "input": {...}, "result_ids": [...]} or
       {"type": "final", "text": ...}."""
    system = ("You are Docent, a Meridian docs assistant. Use the search_docs tool to find relevant pages, "
              "then answer ONLY from what it returns. If the docs don't contain the answer, say "
              "\"I don't have that in the docs.\"")
    messages = [{"role": "user", "content": question}]
    trajectory = []
    for _ in range(max_steps):
        resp = client.messages.create(model=model, max_tokens=400, system=system,
                                       tools=SEARCH_TOOL, messages=messages)
        if resp.stop_reason == "tool_use":
            messages.append({"role": "assistant", "content": resp.content})
            tool_results = []
            for block in resp.content:
                if block.type == "tool_use":
                    hits = search_docs(block.input["query"], docs)
                    trajectory.append({"type": "tool_call", "tool": block.name,
                                       "input": block.input, "result_ids": [h["id"] for h in hits]})
                    tool_results.append({"type": "tool_result", "tool_use_id": block.id,
                                         "content": json.dumps(hits)})
            messages.append({"role": "user", "content": tool_results})
        else:
            text = "".join(b.text for b in resp.content if b.type == "text")
            trajectory.append({"type": "final", "text": text})
            return text, trajectory
    return "(stopped: max steps)", trajectory

Read the loop once more with the failure list from earlier in mind. stop_reason == "tool_use" is the model deciding to search; the block.input["query"] is the arguments it chose; search_docs returns the observation; and the loop repeats max_steps times before it force-stops with "(stopped: max steps)". Every branch in that loop corresponds to a row in the failure list, and the trajectory list is our window into which branch fired when.

Watch the trajectory, not just the answer#

Now run it. We’ll ask three questions and, for each, print the entire trajectory: every search_docs call with the query the agent chose and the doc ids it got back, then the final answer. We deliberately pick a straightforward in-scope question, a second in-scope one, and the out-of-scope MongoDB probe from the golden set — the one whose correct behavior is to refuse.

def show(question):
    answer, trajectory = agentic_docent(question, DOCS)
    n_calls = sum(1 for s in trajectory if s["type"] == "tool_call")
    print("Q:", question)
    for i, step in enumerate(trajectory, 1):
        if step["type"] == "tool_call":
            print(f"  step {i}: TOOL  search_docs(query={step['input']['query']!r}) -> {step['result_ids']}")
        else:
            print(f"  step {i}: FINAL {step['text']}")
    print(f"  ({n_calls} search call(s), {len(trajectory)} step(s) total)")
    print("-" * 70)

for q in ["What does a 401 error mean?",
          "How long are backups retained on the Scale plan?",
          "Does Meridian support MongoDB-style aggregation pipelines?"]:
    show(q)

Q: What does a 401 error mean?
  step 1: TOOL  search_docs(query='401 error') -> ['errors', 'auth']
  step 2: FINAL A **401 error** means a **missing or invalid API key**. 

According to the Meridian documentation, when you receive a 401 error, it indicates that either:
- Your API key is missing from the request, or
- Your API key is invalid

To fix this, make sure you're including a valid API key in the Authorization header as a bearer token. You can create API keys in the Meridian dashboard under Settings > API Keys.
  (1 search call(s), 2 step(s) total)
----------------------------------------------------------------------
Q: How long are backups retained on the Scale plan?
  step 1: TOOL  search_docs(query='Scale plan backup retention') -> ['backups', 'regions']
  step 2: FINAL On the **Scale plan**, automatic daily backups are retained for **90 days**.
  (1 search call(s), 2 step(s) total)
----------------------------------------------------------------------
Q: Does Meridian support MongoDB-style aggregation pipelines?
  step 1: TOOL  search_docs(query='MongoDB aggregation pipelines') -> ['auth', 'ratelimits']
  step 2: TOOL  search_docs(query='aggregation pipelines support') -> ['auth', 'ratelimits']
  step 3: TOOL  search_docs(query='query language features') -> ['auth', 'ratelimits']
  step 4: TOOL  search_docs(query='data processing filters') -> ['regions', 'deletion']
  step 5: TOOL  search_docs(query='Meridian capabilities features') -> ['auth', 'plans']
  step 6: FINAL I don't have that in the docs. The search results don't contain any information about MongoDB-style aggregation pipeline support in Meridian. To get an accurate answer about this feature, I'd recommend checking the main Meridian documentation or contacting their support team directly.
  (5 search call(s), 6 step(s) total)
----------------------------------------------------------------------

Look at what the trajectory reveals that the answer alone would not.

The first two questions are clean single-search-then-answer paths — the shape you want. For the 401 question the agent chose a tight query ('401 error'), got back ['errors', 'auth'] (the errors page has the answer), and answered correctly in one search. The backups question is the same story: one sensible query, the right page (backups), one accurate answer. Two steps each. If you were writing a rubric for “ideal Docent behavior,” this is it.

The MongoDB question is a different animal, and this is where the trajectory earns its keep. The agent searched five times before refusing — 'MongoDB aggregation pipelines', then 'aggregation pipelines support', then 'query language features', then 'data processing filters', then 'Meridian capabilities features' — cycling through the same idea in slightly different words, getting back pages like auth and ratelimits that have nothing to do with the question. The final answer is correct: the docs genuinely don’t cover MongoDB pipelines, and refusing is exactly the right outcome. But the path to that correct outcome was a redundant loop — one of the failure modes from our list — that a final-answer check would score as a clean pass.

That is the whole lesson in one run. Read only the answers and you would record: 401 correct, backups correct, MongoDB correctly refused — a perfect scorecard. Read the trajectories and you find that one of those three “passes” took five tool calls to reach a refusal it could arguably have reached much sooner. The outcome was fine; the process was wasteful. On a live system that is five times the latency and five times the token cost for that class of question, and you would never have seen it.

What the rest of the module does about it#

You have now seen the problem the whole module exists to solve: an agent’s behavior is a trajectory, the trajectory fails in many more ways than an answer does, and you have to grade both the outcome and the process. The remaining lessons give you a metric for each layer:

• Lesson 2 — Task success & outcome metrics. Start with the outcome: did the agent achieve the task? You’ll define success precisely for Docent (right answer, or a correct refusal on the out-of-scope probe) and measure it across the golden set. This is the “outcome” half of the split, done rigorously.
• Lesson 3 — Tool-call correctness. Zoom into a single step: when the agent called search_docs, was it the right tool, and were the arguments valid and sensible? This catches the “wrong tool” and “wrong arguments” failures directly.
• Lesson 4 — Trajectory evaluation. Grade the whole path: number of steps, redundant loops, order, and premature give-ups — exactly the MongoDB-style wandering you just watched. This is the “process” half of the split.
• Lesson 5 — Guided project. Put all three together and evaluate the agentic Docent end to end, so each failure is localized to an outcome problem, a tool-call problem, or a trajectory problem.

Keep the three trajectories from this lesson in mind as you go. Every metric ahead is, at bottom, a way to turn “that path looked off” into a number you can track.

Practice Exercises#

Exercise 1: Count the failure surface#

Take the six failure modes listed in this lesson (wrong tool, wrong arguments, wrong order, redundant loop, gave up early, hit step limit) and, for each of the three trajectories printed above, write down whether that failure occurred, could plausibly have occurred, or was avoided. You don’t need to run code — just annotate the trajectories. The goal is to feel how many independent things had to go right for even the clean 401 answer to be correct.

Exercise 2: Make the trajectory variation visible#

Run the show() loop on the MongoDB question three times in a row and record how many search_docs calls each run makes. Because the agent is non-deterministic, you’ll likely see the count vary. Then run it three times on the 401 question and note whether it varies. This is direct evidence for why an agent metric can’t assert an exact expected transcript.

Exercise 3: Right answer, questionable path#

Write down (in prose or pseudocode) how you would score the MongoDB trajectory if you graded only the final answer, versus if you also graded the process. Then propose a single, concrete process rule that would have flagged this run — for example, “more than 2 search_docs calls for one question is a warning.” Apply your rule to all three trajectories and state which pass and which get flagged.

Summary#

An agent does not produce a single answer; it produces a trajectory — a recorded sequence of tool calls, arguments, observations, and a final response. You turned Docent into a tool-using agent that decides for itself when to call search_docs, ran it live on three questions, and printed the full path of each. The two in-scope questions produced clean single-search-then-answer trajectories (one sensible query, the right page, a correct answer). The out-of-scope MongoDB question produced a correct outcome by a wasteful path: it searched five times in a redundant loop before rightly refusing — a run that a final-answer check scores as a flawless pass while the trajectory shows five times the work it needed. That gap between a right outcome and a questionable process is exactly why agents are harder to evaluate, and why this module grades both.

Key Concepts#

• Trajectory — an agent’s real output is the whole sequence of tool calls, arguments, observations, and the final answer, not just the answer.
• The failure surface multiplies — a single answer is right or wrong; a trajectory can fail by wrong tool, wrong arguments, wrong order, redundant loops, giving up early, or hitting the step limit.
• Outcome vs process — outcome asks “did it achieve the task?”; process asks “was the path sound?”. They come apart: right by luck (bad path, right answer) and perfect path to a wrong answer both happen.
• Final-answer checks are blind to process — a correct-but-wasteful trajectory is indistinguishable from an ideal one if you only read the answer, which is precisely the risk process evaluation exists to expose.
• Agent behavior is non-deterministic — the same question can produce different queries and different step counts run to run, so agent metrics judge the shape of a path, not an exact expected transcript.

Why This Matters#

In production, the agent that is “right by a bad path” is the one that quietly costs you. It passes your outcome tests, so it ships; then it burns latency and tokens on redundant loops, and on the next slightly different question the fragile path finally breaks — with no warning, because the answers looked fine all along. Being able to read and score a trajectory turns “the agent seems slow and flaky sometimes” into a specific, measurable claim: this class of question triggers a five-call redundant loop before refusing. That is the difference between guessing at an agent’s reliability and engineering it, and it is only possible once you evaluate the process alongside the outcome — which is what the rest of this module teaches you to do.

Continue Building Your Skills#

You’ve watched a single Docent answer become a whole path of tool calls, and seen that even a correct answer can hide a wasteful or fragile trajectory that a final-answer check would wave through. Knowing that a trajectory can go wrong in six different places is the first step; the next is putting numbers on it, one layer at a time. In Lesson 2 you’ll pin down the outcome layer — a precise, measurable definition of task success for the agentic Docent — so that “did it achieve the goal?” stops being a judgment call and becomes a score you can track across every question in the golden set. From there the module builds outward to tool-call correctness and full trajectory evaluation, until you can look at any run and say exactly which layer failed.