Lesson 4 - Trajectory Evaluation

Welcome to Trajectory Evaluation#

In Lesson 2 you asked whether Docent – our agent for the Meridian serverless database docs – accomplished the task, and in Lesson 3 you asked whether each individual tool call was correct: the right tool, valid arguments, a sensible query. Both are essential, and both share a blind spot. They look at the endpoints – the final answer, or one call at a time – and never at the shape of the whole run.

Here’s the problem that shape hides. Ask two versions of Docent the same question and both come back with “HTTP 429.” One of them searched the docs once and answered. The other searched, ignored what came back, searched again with a barely-different query, searched a third time, and only then answered. Same outcome. Every individual tool call was a valid search_docs call with reasonable arguments, so a per-call check passes each one. Yet the second agent burned three times the calls, latency, and tokens, and wandered in a loop it was lucky to escape. Efficiency, redundancy, and robustness don’t live in the answer or in any single call – they live in the trajectory, the ordered sequence of steps from question to final answer.

This lesson evaluates that sequence. You’ll build cheap, deterministic metrics over a recorded trajectory, then add an LLM judge for the open-ended case where “was this process sensible?” needs judgement, not a rule.

In this lesson, you will:

• See why the path matters: two agents can both succeed while one is efficient and one wanders
• Run the agentic Docent live and collect a real trajectory of steps
• Build deterministic trajectory metrics – num_tool_calls, has_redundant_call, and matches_reference against an expected path – and prove they’re reproducible
• Score a clean one-search trajectory against a hand-built wasteful one and watch the metrics separate them
• Add a live claude-haiku-4-5 trajectory judge that rates process quality 1-5 with a reason, for open-ended paths where no single reference exists

Why the path matters#

Outcome evaluation asks did you get there? Trajectory evaluation asks how did you get there? – and the “how” is where an agent’s real cost and fragility hide.

Think about what a bad-but-successful path actually costs. Every extra tool call is another model round-trip: more latency for the user, more tokens on the bill, and more chances for something downstream to fail. An agent that loops – issuing the same search over and over, hoping for a different result – is one prompt tweak away from spinning until it hits the step limit and gives up. And an agent that stops too early, answering before it has searched, is confidently ungrounded. None of these are visible in the final string if the agent happened to land the right answer anyway. They only show up when you look at the sequence.

So what do we look for in a trajectory? A few concrete signals cover most of what goes wrong:

• Efficiency – how many steps did it take? For a simple factual lookup, one search and an answer is ideal; five steps for the same question is a smell.
• Redundancy / loops – did it repeat itself, calling the same tool with a near-identical query? Repetition means the agent isn’t using what it already retrieved.
• Recovery – when a search came back empty or off-target, did the agent adapt (reformulate, try a different angle) or did it stall?
• Appropriate stopping – did it stop when it had enough (not spinning to the step limit), and not stop before it had searched at all (not giving up early)?

Two ways to check these. When you know the path a task should take – and for a docs lookup you often do, it’s “search once, then answer” – you compare the actual trajectory to that reference trajectory, exactly or step by step. When the right path is open-ended (a multi-hop question where several routes are reasonable), there’s no single reference to match, so you hand the whole trajectory to an LLM judge and ask it to rate the process. We’ll build both, deterministic first.

Collecting a real trajectory#

The agentic Docent from earlier in this module decides for itself when to search. Crucially, it records a trajectory as it runs: a list of step dicts, each either a tool_call (with the query it sent and the page ids that came back) or the final answer. That recording is what we evaluate. Here is the agent, unchanged from the module’s canonical version:

import anthropic, json
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

# DOCS is the 8-page Meridian corpus defined earlier in the course (auth, ratelimits,
# plans, regions, errors, backups, sdks, deletion).

SEARCH_TOOL = [{
    "name": "search_docs",
    "description": "Search the Meridian documentation and return matching pages. Call this before answering.",
    "input_schema": {
        "type": "object",
        "properties": {"query": {"type": "string", "description": "search query"}},
        "required": ["query"],
    },
}]

def search_docs(query, docs, k=2):
    q = set(query.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())), reverse=True)
    hits = scored[:k]
    return [{"id": d["id"], "title": d["title"], "text": d["text"]} for d in hits]

def agentic_docent(question, docs, model="claude-haiku-4-5", max_steps=4):
    """Returns (final_answer, trajectory). Each trajectory step is a dict:
       {"type": "tool_call", "tool": ..., "input": {...}, "result_ids": [...]} or
       {"type": "final", "text": ...}."""
    system = ("You are Docent, a Meridian docs assistant. Use the search_docs tool to find relevant pages, "
              "then answer ONLY from what it returns. If the docs don't contain the answer, say "
              "\"I don't have that in the docs.\"")
    messages = [{"role": "user", "content": question}]
    trajectory = []
    for _ in range(max_steps):
        resp = client.messages.create(model=model, max_tokens=400, system=system,
                                       tools=SEARCH_TOOL, messages=messages)
        if resp.stop_reason == "tool_use":
            messages.append({"role": "assistant", "content": resp.content})
            tool_results = []
            for block in resp.content:
                if block.type == "tool_use":
                    hits = search_docs(block.input["query"], docs)
                    trajectory.append({"type": "tool_call", "tool": block.name,
                                       "input": block.input, "result_ids": [h["id"] for h in hits]})
                    tool_results.append({"type": "tool_result", "tool_use_id": block.id,
                                         "content": json.dumps(hits)})
            messages.append({"role": "user", "content": tool_results})
        else:
            text = "".join(b.text for b in resp.content if b.type == "text")
            trajectory.append({"type": "final", "text": text})
            return text, trajectory
    return "(stopped: max steps)", trajectory

Run it once on a simple factual question and print the trajectory it recorded. Because this is a live model deciding its own steps, treat the exact steps as a real recorded run – yours will be close but may word the query differently:

def signature(traj):
    """A compact one-line-per-step view of a trajectory."""
    out = []
    for step in traj:
        if step["type"] == "tool_call":
            out.append(f'search_docs(query={step["input"]["query"]!r}) -> {step["result_ids"]}')
        else:
            out.append(f'final: {step["text"][:70]}')
    return out

question = "What HTTP status code does Meridian return when you exceed the rate limit?"
answer, traj = agentic_docent(question, DOCS)

print("Task:", question)
print("-" * 74)
for i, line in enumerate(signature(traj), 1):
    print(f"step {i}: {line}")
print("-" * 74)
print("final answer:", answer)

Task: What HTTP status code does Meridian return when you exceed the rate limit?
--------------------------------------------------------------------------
step 1: search_docs(query='rate limit HTTP status code') -> ['errors', 'ratelimits']
step 2: final: Meridian returns HTTP status code **429** when you exceed the rate lim
--------------------------------------------------------------------------
final answer: Meridian returns HTTP status code **429** when you exceed the rate limit. 

When this happens, the response includes a `Retry-After` header that indicates how many seconds you should wait before retrying your request.

That’s a clean trajectory: one search with a sensible query, the right pages retrieved, then an answer. This is what “good process” looks like for a lookup. Now we need metrics that can say it’s good – and flag a trajectory that isn’t.

Deterministic trajectory metrics#

The signals from the first section – efficiency, redundancy, reference-match – are computable directly from the recorded steps, with no model in the loop. That makes them cheap, instant, and perfectly reproducible, which is exactly what you want for a metric you’ll run over thousands of trajectories in CI.

To make the comparison sharp, we score two trajectories in the same recorded shape. CLEAN is the live run from above: one search, then a final answer. WASTEFUL is hand-constructed to fail on process while reaching the identical answer – it searches, ignores the hit, searches again with a near-identical query, searches a third time, and only then answers. Same outcome, wandering path:

CLEAN = [
    {"type": "tool_call", "tool": "search_docs",
     "input": {"query": "rate limit HTTP status code"}, "result_ids": ["errors", "ratelimits"]},
    {"type": "final", "text": "Meridian returns HTTP 429 when you exceed the rate limit."},
]

WASTEFUL = [
    {"type": "tool_call", "tool": "search_docs",
     "input": {"query": "rate limit status code"}, "result_ids": ["errors", "ratelimits"]},
    {"type": "tool_call", "tool": "search_docs",
     "input": {"query": "rate limit HTTP status code"}, "result_ids": ["errors", "ratelimits"]},
    {"type": "tool_call", "tool": "search_docs",
     "input": {"query": "what code for rate limit"}, "result_ids": ["ratelimits", "errors"]},
    {"type": "final", "text": "Meridian returns HTTP 429 when you exceed the rate limit."},
]

# The expected path for a simple factual question: one search, then a final answer.
REFERENCE = ["tool_call:search_docs", "final"]

Now the three metrics. num_tool_calls is raw efficiency. has_redundant_call catches loops by comparing every pair of same-tool calls: if two queries share enough tokens (Jaccard overlap over a threshold), the agent essentially asked the same thing twice. matches_reference reduces the trajectory to its sequence of step kinds and checks it exactly against the expected path:

def num_tool_calls(traj):
    """How many tool calls the agent made -- the core efficiency signal."""
    return sum(1 for s in traj if s["type"] == "tool_call")

def _query_similarity(a, b):
    """Jaccard token overlap between two queries (0.0 = disjoint, 1.0 = same tokens)."""
    ta, tb = set(a.lower().split()), set(b.lower().split())
    return len(ta & tb) / len(ta | tb) if (ta | tb) else 0.0

def has_redundant_call(traj, threshold=0.5):
    """True if two calls to the same tool used near-identical queries (a wasted step / loop)."""
    calls = [s for s in traj if s["type"] == "tool_call"]
    for i in range(len(calls)):
        for j in range(i + 1, len(calls)):
            if calls[i]["tool"] == calls[j]["tool"]:
                if _query_similarity(calls[i]["input"]["query"], calls[j]["input"]["query"]) >= threshold:
                    return True
    return False

def step_signature(traj):
    """Reduce a trajectory to its step kinds, e.g. ['tool_call:search_docs', 'final']."""
    sig = []
    for s in traj:
        sig.append(f'{s["type"]}:{s["tool"]}' if s["type"] == "tool_call" else s["type"])
    return sig

def matches_reference(traj, reference=REFERENCE):
    """Exact match of the step sequence against the expected path."""
    return step_signature(traj) == reference

def score(traj):
    return {
        "num_tool_calls": num_tool_calls(traj),
        "has_redundant_call": has_redundant_call(traj),
        "matches_reference": matches_reference(traj),
    }

print("Deterministic trajectory metrics")
print("-" * 62)
for name, traj in [("CLEAN", CLEAN), ("WASTEFUL", WASTEFUL)]:
    m = score(traj)
    print(f"{name:<10} tool_calls={m['num_tool_calls']}  "
          f"redundant={str(m['has_redundant_call']):<5}  "
          f"matches_reference={m['matches_reference']}")

Deterministic trajectory metrics
--------------------------------------------------------------
CLEAN      tool_calls=1  redundant=False  matches_reference=True
WASTEFUL   tool_calls=3  redundant=True   matches_reference=False

Every metric separates the two, and none of them ever looked at the answer – which is identical for both. CLEAN takes one call, has no repeat, and matches the expected search-then-answer path exactly. WASTEFUL triples the call count, trips the redundancy check (its first two queries share “rate limit … status code” well over the 0.5 threshold), and fails the reference match because its signature is three tool calls, not one. This is the whole point of trajectory evaluation: a difference that outcome and per-call checks are structurally blind to becomes three crisp, red flags.

Because there’s no model in the loop, these metrics are reproducible. Run the same block again and the numbers are byte-identical:

print("Deterministic trajectory metrics")
print("-" * 62)
for name, traj in [("CLEAN", CLEAN), ("WASTEFUL", WASTEFUL)]:
    m = score(traj)
    print(f"{name:<10} tool_calls={m['num_tool_calls']}  "
          f"redundant={str(m['has_redundant_call']):<5}  "
          f"matches_reference={m['matches_reference']}")

Deterministic trajectory metrics
--------------------------------------------------------------
CLEAN      tool_calls=1  redundant=False  matches_reference=True
WASTEFUL   tool_calls=3  redundant=True   matches_reference=False

An LLM judge for open-ended trajectories#

Deterministic metrics are ideal when you can name the right path, but they can’t tell you whether an unfamiliar trajectory was sensible. Was reformulating the query the second time a smart recovery or a wasteful loop? Did stopping after two searches show good judgement or a premature give-up? When there’s no single reference path to match, this is exactly the open-ended judgement call an LLM judge is good at – the same pattern from Module 4, now pointed at the process instead of the answer.

The judge reads the task and the full rendered trajectory, then returns a 1-5 process-quality score with a one-sentence reason as JSON. Note the prompt tells it what “good process” means – efficient, recovers, stops appropriately – so it grades against that, not its own private idea of good:

import json, re
import anthropic
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

def render_trajectory(traj):
    """Turn a recorded trajectory into readable text for the judge."""
    lines = []
    for i, s in enumerate(traj, 1):
        if s["type"] == "tool_call":
            lines.append(f'step {i}: search_docs(query="{s["input"]["query"]}") '
                         f'returned pages {s["result_ids"]}')
        else:
            lines.append(f'step {i}: final answer: {s["text"]}')
    return "\n".join(lines)

def extract_json(text):
    m = re.search(r"\{.*\}", text, re.DOTALL)
    return m.group(0) if m else None

def judge_trajectory(task, traj, model="claude-haiku-4-5"):
    prompt = (
        "You are evaluating the PROCESS an AI agent took to answer a task, not just its final answer. "
        "A good process is efficient (no wasted or repeated searches), recovers from bad results, and "
        "stops at the right time. Rate the process quality from 1 (wasteful or broken) to 5 (clean and "
        "efficient).\n\n"
        f"Task: {task}\n\n"
        f"Agent trajectory:\n{render_trajectory(traj)}\n\n"
        'Respond with a single JSON object and nothing else: '
        '{"score": <integer 1-5>, "reason": "<one sentence>"}'
    )
    msg = client.messages.create(model=model, max_tokens=150,
                                 messages=[{"role": "user", "content": prompt}])
    return json.loads(extract_json(msg.content[0].text))

task = "What HTTP status code does Meridian return when you exceed the rate limit?"

print("Live trajectory judge (claude-haiku-4-5)")
print("=" * 62)
for name, traj in [("CLEAN", CLEAN), ("WASTEFUL", WASTEFUL)]:
    verdict = judge_trajectory(task, traj)
    print(f"\n[{name}] score={verdict['score']}/5")
    print(f"  reason: {verdict['reason']}")

Live trajectory judge (claude-haiku-4-5)
==============================================================

[CLEAN] score=5/5
  reason: The agent efficiently identified relevant documentation pages on the first search and provided a specific, confident answer without unnecessary steps or redundant searches.

[WASTEFUL] score=3/5
  reason: The agent found relevant pages quickly but made redundant searches (steps 1-3 were essentially identical queries) before providing an answer, though it did eventually reach the correct conclusion without further backtracking.

The judge agrees with the deterministic metrics and adds something they can’t: a reason in plain language. It gives the clean path a 5 and names why – one search, no waste. It docks the wasteful path to a 3 and names the exact flaw: “redundant searches … essentially identical queries.” That reasoning is the reason to reach for a judge on open-ended trajectories, where you couldn’t have written a rule for “was reformulating here smart or dumb?” in advance.

Be honest about variation, though. The judge is a live model, so its scores wobble a little run to run: re-run this and the clean trajectory might come back a 4 instead of a 5 (one run wished it had “explicitly verified … the ratelimits page content”), while the wasteful one holds at 3. The absolute number drifts by a level on the margin; what stays stable is the ranking – clean above wasteful, every run. That’s the same lesson as the rubric judge in Module 4, and the same reason the deterministic metrics around it are worth keeping: num_tool_calls and has_redundant_call never drift, so let them carry the objective signal and use the judge for the qualitative “was this sensible?” that only judgement can answer.

Practice Exercises#

Exercise 1: Add a “gave up early” metric#

An agent that answers without ever searching is a distinct trajectory failure – it’s ungrounded by construction. Write answered_without_search(traj) that returns True when the trajectory’s only step is a final (no tool_call precedes it). Build a third trajectory NO_SEARCH that jumps straight to a final answer, and score all three trajectories with your new metric plus the existing ones.

Exercise 2: Make matches_reference tolerant with step-level overlap#

Exact matching punishes a fine trajectory that happens to search twice. Write reference_overlap(traj, reference) that returns the fraction of the reference signature covered by the trajectory’s signature – e.g. how many of ["tool_call:search_docs", "final"]’s elements appear in the actual step signature. Report it for CLEAN and WASTEFUL and compare to the strict boolean.

Exercise 3: Judge a live multi-step trajectory#

The “Free plan” question exposes the retrieval weakness in search_docs (keyword overlap can miss the plans page), which sometimes makes the live agent search more than once. Run agentic_docent("What does Meridian charge for the Free plan?", DOCS) a few times, keep a run whose trajectory has two or more tool calls, and pass it to judge_trajectory. Does the judge’s score and reason match what your deterministic metrics say about that same real trajectory?

Summary#

Trajectory evaluation grades the process – the ordered sequence of steps an agent takes from question to answer – and it catches what outcome checks (Lesson 2) and per-call checks (Lesson 3) are structurally blind to. Two Docent runs can reach the identical answer while one takes a single clean search and the other loops through three near-identical searches; the difference lives only in the trajectory. You ran the agentic Docent live, collected a real recorded trajectory, and built three deterministic, reproducible metrics over it: num_tool_calls for efficiency, has_redundant_call to catch loops via query-token overlap, and matches_reference for exact comparison against the expected “one search, then answer” path. Those metrics gave the clean trajectory 1 / False / True and the wasteful one 3 / True / False, byte-identical on re-run. For open-ended paths where no single reference exists, you added a live claude-haiku-4-5 trajectory judge that rated the clean path 5/5 and the wasteful path 3/5 with a plain-language reason naming the redundant searches – honest that the score wobbles a level run to run while the ranking holds.

Key Concepts#

• Trajectory – the ordered sequence of steps (tool calls plus the final answer) an agent takes; the unit that trajectory evaluation grades.
• Process vs outcome – two agents can produce the same answer by very different paths; efficiency, redundancy, and robustness live in the path, not the answer.
• num_tool_calls – raw step-count efficiency; a two-sided signal, since both too many (looping) and zero (giving up early) are failures.
• has_redundant_call – detects loops by comparing same-tool queries with a token-overlap threshold; repetition means the agent isn’t using what it already retrieved.
• Reference-trajectory match – comparing the actual step sequence to an expected path, strictly (exact match) for narrow paths or via step-level overlap for open ones.
• Trajectory judge – an LLM rating process quality 1-5 with a reason, for open-ended paths where no single reference exists; deterministic in ranking, wobbly by a level in absolute score.

Why This Matters#

In production, the answer is only half of what an agent costs you. Every wasted tool call is latency the user feels and tokens you pay for, and a loop is one bad prompt away from spinning to the step limit and failing outright – yet a final-answer eval sails right past all of it whenever the agent happens to land the right answer. Trajectory metrics are how you put a number on that hidden cost and gate against it: you can alert when the average tool-call count creeps up after a prompt change, block a deploy that starts producing redundant loops, or slice your eval set to find the topics where the agent wanders. The split you built here – cheap deterministic metrics carrying the objective signal (call counts, loops, reference match) and an LLM judge handling the qualitative “was this path sensible?” – is exactly how mature agent evaluation is structured, because each covers the other’s blind spot. With outcome, tool-call correctness, and now trajectory in hand, you have all three levels an agent needs. The guided project next assembles them into a single evaluation of Docent as a real tool-using agent.

Continue Building Your Skills#

You can now evaluate an agent’s whole path, not just its answer. You collected a real Docent trajectory, then scored it with deterministic metrics that need no model – num_tool_calls for efficiency, has_redundant_call for loops, matches_reference for the expected path – watching a clean one-search run and a wasteful three-search run reach the same answer while every trajectory metric separated them. Then you added a live claude-haiku-4-5 judge for the open-ended case, rating the clean path 5/5 and the wasteful one 3/5 with a reason you could read and check, honest about the run-to-run wobble in the score and the stability in the ranking. That completes the three levels of agent evaluation this module set out to teach: task success (did it get there?), tool-call correctness (was each call valid?), and trajectory (was the path efficient and sound?). The guided project in Lesson 5 brings all three together, turning Docent loose on the golden Q&A set and evaluating its outcomes, its tool calls, and its trajectories in one harness – the closest thing yet to how you’d evaluate a real tool-using agent before shipping it.