Lesson 5 - Guided Project: Evaluating an Agentic Docent

Welcome to the Guided Project#

This is the capstone for Evaluating Agents & Tool Use, and it is where the three levels you learned in this module finally run together on one agent. Over the last four lessons you saw that an agent is not just an answer — it is a path: the agentic Docent decides on its own to call search_docs, reads what comes back, maybe searches again, and only then replies. That path is a new surface to evaluate, and a single failure can hide in any of three places. Task success (Lesson 2) asks whether the final answer was actually right. Tool-call correctness (Lesson 3) asks whether the agent called the right tool with valid, sensible arguments. Trajectory evaluation (Lesson 4) asks whether the sequence of steps was efficient and sound, or whether the agent wandered. In this project you wire all three into a single harness and turn it on Docent.

The deliverable is an agent scorecard: one success rate, one mean tool-call-correctness score, and one mean trajectory-quality score, plus a short list of exactly which tasks to fix and why. What makes the scorecard worth building is that the three numbers disagree on purpose — an agent can ace outcome and tool calls yet still lose points on trajectory, because “it got the right answer” and “it got there the right way” are different claims. You will watch that happen live: Docent answers all eight tasks correctly, makes valid tool calls every time, and still leaves the trajectory level flagging a task where it reached the correct refusal only after searching three times. We keep the live agent runs small and honest about run-to-run variation, then do all the scoring deterministically over the collected trajectories so the numbers reproduce exactly.

By the end of this project, you will be able to:

• Assemble an agent task set from the golden Q&A — factual tasks plus the out-of-scope refusal — each with a machine-checkable success criterion
• Run the tool-using Docent live on every task and collect a structured record: the final answer plus the full recorded trajectory of tool calls
• Score all three levels — task success, tool-call correctness, and trajectory metrics — deterministically over the collected runs, and confirm the numbers are identical on a re-run
• Assemble an agent scorecard and read it: separate “right answer” from “right path,” spot where the agent succeeded down a poor path, and say what you’d fix

Stage 1: The Agent and the Task Set#

An agent evaluation needs two ingredients: the agent under test, and a set of tasks each paired with a success criterion — a machine-checkable rule for whether the final answer counts as a pass. We reuse the module’s agentic_docent verbatim: it runs a real tool-use loop over the eight Meridian docs, calling search_docs when it decides to, and — crucially — it records a trajectory, a list of every tool call with the query it sent and the doc ids that came back. That recorded trajectory is what makes all three levels scoreable later.

For tasks we draw eight questions from the golden set: seven factual ones spanning rate limits, plans, regions, backups, and SDKs, plus the out-of-scope probe (“MongoDB-style aggregation pipelines”) whose only correct behavior is to refuse. Each task carries a verify function — a tiny deterministic verifier over the final answer, exactly the Level-2 outcome check from earlier in the module. The free-cap task is deliberately the Free-plan rate-limit question phrased the roundabout way a user might type it; it’s the one most exposed to the keyword retriever’s weakness, so it’s the task to watch.

import warnings; warnings.filterwarnings("ignore")
import anthropic, json

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

DOCS = [
    {"id": "auth", "title": "Authentication",
     "text": "Meridian authenticates every request with an API key sent in the Authorization header as a bearer token. "
             "Create keys in the dashboard under Settings > API Keys. Keys are either test or live mode; test keys only "
             "touch sandbox data. A key is shown once at creation and cannot be recovered, only rotated."},
    {"id": "ratelimits", "title": "Rate Limits",
     "text": "The Free plan allows 60 requests per minute. The Pro plan allows 600 requests per minute. Exceeding the "
             "limit returns HTTP 429 with a Retry-After header giving the seconds to wait. Rate limits are applied per "
             "API key, not per account."},
    {"id": "plans", "title": "Plans and Pricing",
     "text": "Meridian has three plans. Free costs 0 dollars per month and includes 1 GB of storage. Pro costs 25 dollars "
             "per month and includes 50 GB of storage. Scale costs 199 dollars per month and includes 500 GB of storage. "
             "All plans include unlimited read replicas."},
    {"id": "regions", "title": "Regions",
     "text": "Databases can be created in three regions: us-east, eu-west, and ap-south. The region is fixed at creation "
             "time and cannot be changed later; to move data to another region you must export and re-import into a new "
             "database. Cross-region read replicas are available on the Scale plan only."},
    {"id": "errors", "title": "Error Codes",
     "text": "Meridian uses standard HTTP status codes. 400 means a malformed request, 401 means a missing or invalid API "
             "key, 403 means the key is valid but lacks permission, 404 means the resource does not exist, and 429 means "
             "the rate limit was exceeded. 5xx codes indicate a server-side error and should be retried with backoff."},
    {"id": "backups", "title": "Backups",
     "text": "Automatic daily backups are retained for 7 days on the Free plan, 30 days on Pro, and 90 days on Scale. "
             "Backups run at 03:00 UTC. Point-in-time recovery to any second in the retention window is available on the "
             "Scale plan. Restoring a backup creates a new database and never overwrites the original."},
    {"id": "sdks", "title": "SDKs",
     "text": "Official SDKs are available for Python, JavaScript, and Go. Install the Python SDK with 'pip install meridian'. "
             "The SDK reads the MERIDIAN_API_KEY environment variable by default. All SDKs retry idempotent requests up to "
             "3 times on 5xx errors with exponential backoff."},
    {"id": "deletion", "title": "Deleting Data",
     "text": "Deleting a database is immediate and irreversible once the 24-hour grace period ends. During the grace period "
             "the database is soft-deleted and can be restored from the dashboard. After 24 hours the data and its backups "
             "are permanently purged. Deletion requires a live-mode key with admin permission."},
]

# --- the agent under test: the module's tool-using Docent (records a trajectory) ---
SEARCH_TOOL = [{
    "name": "search_docs",
    "description": "Search the Meridian documentation and return matching pages. Call this before answering.",
    "input_schema": {
        "type": "object",
        "properties": {"query": {"type": "string", "description": "search query"}},
        "required": ["query"],
    },
}]

def search_docs(query, docs, k=2):
    q = set(query.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())), reverse=True)
    hits = scored[:k]
    return [{"id": d["id"], "title": d["title"], "text": d["text"]} for d in hits]

def agentic_docent(question, docs, model="claude-haiku-4-5", max_steps=4):
    system = ("You are Docent, a Meridian docs assistant. Use the search_docs tool to find relevant pages, "
              "then answer ONLY from what it returns. If the docs don't contain the answer, say "
              "\"I don't have that in the docs.\"")
    messages = [{"role": "user", "content": question}]
    trajectory = []
    for _ in range(max_steps):
        resp = client.messages.create(model=model, max_tokens=400, system=system,
                                       tools=SEARCH_TOOL, messages=messages)
        if resp.stop_reason == "tool_use":
            messages.append({"role": "assistant", "content": resp.content})
            tool_results = []
            for block in resp.content:
                if block.type == "tool_use":
                    hits = search_docs(block.input["query"], docs)
                    trajectory.append({"type": "tool_call", "tool": block.name,
                                       "input": block.input, "result_ids": [h["id"] for h in hits]})
                    tool_results.append({"type": "tool_result", "tool_use_id": block.id,
                                         "content": json.dumps(hits)})
            messages.append({"role": "user", "content": tool_results})
        else:
            text = "".join(b.text for b in resp.content if b.type == "text")
            trajectory.append({"type": "final", "text": text})
            return text, trajectory
    return "(stopped: max steps)", trajectory

# --- verifiers: the Level-2 outcome check, one per task ---
def contains(*subs):
    def check(ans):
        a = ans.lower()
        return all(s.lower() in a for s in subs)
    return check

def refuses(ans):
    return "don't have that in the docs" in ans.lower() or "i don't have" in ans.lower()

# --- the task set: 7 factual golden questions + 1 out-of-scope refusal ---
TASKS = [
    dict(id="rate-429",   q="What HTTP status code does Meridian return when you exceed the rate limit?",
         doc="ratelimits", verify=contains("429")),
    dict(id="pro-rpm",    q="How many requests per minute does the Pro plan allow?",
         doc="ratelimits", verify=contains("600")),
    dict(id="pro-cost",   q="How much does the Pro plan cost per month?",
         doc="plans",      verify=contains("25")),
    dict(id="region",     q="Can you change a database's region after it is created?",
         doc="regions",    verify=lambda a: ("no" in a.lower() or "cannot" in a.lower() or "can't" in a.lower())
                                            and not refuses(a)),
    dict(id="scale-bkup", q="How long are backups retained on the Scale plan?",
         doc="backups",    verify=contains("90")),
    dict(id="sdk-env",    q="Which environment variable does the Python SDK read for the API key?",
         doc="sdks",       verify=contains("MERIDIAN_API_KEY")),
    dict(id="free-cap",   q="What is the per-minute request cap on the Free plan pricing tier?",
         doc="ratelimits", verify=contains("60")),
    dict(id="oos-mongo",  q="Does Meridian support MongoDB-style aggregation pipelines?",
         doc=None,         verify=refuses),
]

print(f"Task set: {len(TASKS)} tasks ({sum(1 for t in TASKS if t['doc'])} factual, "
      f"{sum(1 for t in TASKS if t['doc'] is None)} out-of-scope refusal)")
for t in TASKS:
    print(f"  {t['id']:11} doc={str(t['doc']):11} {t['q']}")

Task set: 8 tasks (7 factual, 1 out-of-scope refusal)
  rate-429    doc=ratelimits  What HTTP status code does Meridian return when you exceed the rate limit?
  pro-rpm     doc=ratelimits  How many requests per minute does the Pro plan allow?
  pro-cost    doc=plans       How much does the Pro plan cost per month?
  region      doc=regions     Can you change a database's region after it is created?
  scale-bkup  doc=backups     How long are backups retained on the Scale plan?
  sdk-env     doc=sdks        Which environment variable does the Python SDK read for the API key?
  free-cap    doc=ratelimits  What is the per-minute request cap on the Free plan pricing tier?
  oos-mongo   doc=None        Does Meridian support MongoDB-style aggregation pipelines?

Every task carries the one field the agent doesn’t get to see: doc, the id of the Meridian page that actually contains the answer (None for the out-of-scope probe). We never feed it to the agent — the agent must find it by searching — but we keep it on the side because the trajectory level needs it to ask “did the agent’s own search ever surface the right page?” That single label is what lets us tell a lucky answer from a well-retrieved one.

Stage 2: Run the Agent Live and Collect#

Now we turn the agent loose. For every task we call agentic_docent live on claude-haiku-4-5 and store a run record: the task, the agent’s final answer, and its full trajectory. Because the agent controls its own steps, this is the one live, non-deterministic part of the project — the exact wording and even the number of searches can shift between runs. We collect everything once here so that all the scoring in Stage 3 reads from a fixed, in-memory set and stays reproducible.

Below we print three trajectories in full: a clean single-search task, the exposed free-cap task, and the out-of-scope probe. Read them as paths, not just answers — the tool call, the doc ids it returned, and the final reply are each a place something could go right or wrong.

# run the agent live on every task; collect (answer, trajectory) per task
runs = []
for t in TASKS:
    answer, trajectory = agentic_docent(t["q"], DOCS)
    runs.append(dict(task=t, answer=answer, trajectory=trajectory))

def show_trajectory(run):
    print(f"Q: {run['task']['q']}")
    for step in run["trajectory"]:
        if step["type"] == "tool_call":
            print(f"  [tool] search_docs(query={step['input']['query']!r}) -> {step['result_ids']}")
        else:
            print(f"  [final] {step['text']}")
    print()

print("--- sample trajectories ---")
show_trajectory(runs[0])                                                   # a clean single-search task
show_trajectory(next(r for r in runs if r["task"]["id"] == "free-cap"))    # the exposed Free-plan task
show_trajectory(next(r for r in runs if r["task"]["id"] == "oos-mongo"))   # the out-of-scope probe

--- sample trajectories ---
Q: What HTTP status code does Meridian return when you exceed the rate limit?
  [tool] search_docs(query='rate limit HTTP status code') -> ['errors', 'ratelimits']
  [final] When you exceed the rate limit in Meridian, the API returns **HTTP 429** (Too Many Requests). 

Additionally, the response includes a **Retry-After** header that indicates the number of seconds you should wait before making another request.

Q: What is the per-minute request cap on the Free plan pricing tier?
  [tool] search_docs(query='Free plan pricing tier per-minute request cap') -> ['ratelimits', 'plans']
  [final] The Free plan allows **60 requests per minute**. Rate limits are applied per API key, not per account.

Q: Does Meridian support MongoDB-style aggregation pipelines?
  [tool] search_docs(query='MongoDB aggregation pipelines') -> ['auth', 'ratelimits']
  [tool] search_docs(query='aggregation pipeline query') -> ['auth', 'ratelimits']
  [tool] search_docs(query='Meridian query capabilities features') -> ['auth', 'plans']
  [final] I don't have that in the docs. The search results don't contain information about whether Meridian supports MongoDB-style aggregation pipelines. You may want to check the full Meridian documentation or contact their support team for details.

Three very different paths to compare. The rate-429 task is textbook: one search, the ratelimits page comes back, the agent answers correctly — the reference trajectory. The free-cap task is the interesting one: recall from Module 5 that this exact question broke the single-pass Docent because keyword retrieval fetched the wrong pages. Here the agent rescued it — it wrote its own query, 'Free plan pricing tier per-minute request cap', packed with enough overlapping tokens that ratelimits came back first, and it answered “60 requests per minute” correctly. Letting the model author the search query papered over the retriever’s weakness. And the out-of-scope probe reached the right outcome — a refusal — but only after three searches, hunting for a page that doesn’t exist before giving up. That’s a correct answer down a wasteful path, and it’s exactly what the trajectory level exists to catch.

Because this is a live agent, your re-run won’t be byte-identical: the wording will shift, and the out-of-scope probe might refuse after two searches instead of three. The outcomes are stable — the facts and the refusal hold — but the path length wobbles, which is precisely why we score the path.

Stage 3: Score All Three Levels#

Everything from here is deterministic. We don’t call the model again — we score the trajectories we already collected, so the numbers reproduce byte-for-byte. We compute three levels per task and assemble one combined row each.

Level 1 — task success: run the task’s verify on the final answer.

Level 2 — tool-call correctness: called_search (did it use the tool at all), args_valid (was every query a non-empty string), and query_quality (a 0-1 overlap between the agent’s query terms and the question’s content words — a cheap proxy for “was the search sensible”).

Level 3 — trajectory metrics: num_tool_calls (path length), redundancy (identical queries fired more than once — wasted steps), and matches_reference (did the path match the ideal shape: exactly one search that surfaced the relevant doc, or for the out-of-scope task exactly one search then a refusal).

def tool_calls(run):
    return [s for s in run["trajectory"] if s["type"] == "tool_call"]

# Level 1 -- task success (verifier over the final answer)
def task_success(run):
    return bool(run["task"]["verify"](run["answer"]))

# Level 2 -- tool-call correctness
def called_search(run):
    return any(tc["tool"] == "search_docs" for tc in tool_calls(run))

def args_valid(run):
    tcs = tool_calls(run)
    return bool(tcs) and all(isinstance(tc["input"].get("query"), str) and tc["input"]["query"].strip()
                             for tc in tcs)

STOP = {"the","a","an","of","to","is","are","do","does","how","what","which","can","you","on","in",
        "for","per","it","that","and","when"}
def query_quality(run):
    tcs = tool_calls(run)
    if not tcs:
        return 0.0
    qwords = {w for w in run["task"]["q"].lower().replace("?","").replace("'","").split() if w not in STOP}
    scores = []
    for tc in tcs:
        terms = {w for w in tc["input"]["query"].lower().split() if w not in STOP}
        scores.append(len(qwords & terms) / len(terms) if terms else 0.0)
    return round(sum(scores) / len(scores), 2)

# Level 3 -- trajectory metrics
def num_tool_calls(run):
    return len(tool_calls(run))

def redundancy(run):                          # identical queries fired more than once
    qs = [tc["input"]["query"].strip().lower() for tc in tool_calls(run)]
    return len(qs) - len(set(qs))

def matches_reference(run):                   # ideal shape: one search that surfaces the relevant doc
    tcs = tool_calls(run)
    if len(tcs) != 1:
        return False
    if run["task"]["doc"] is None:
        return True                           # out-of-scope: one search then refuse
    return run["task"]["doc"] in tcs[0]["result_ids"]

def score_all(runs):
    rows = []
    for run in runs:
        rows.append(dict(
            id=run["task"]["id"],  success=task_success(run),
            called=called_search(run), valid=args_valid(run), qquality=query_quality(run),
            ncalls=num_tool_calls(run), redund=redundancy(run), matches=matches_reference(run),
        ))
    return rows

rows  = score_all(runs)
rows2 = score_all(runs)   # deterministic scoring over the fixed collected runs

print(f"{'task':11}{'succ':>5}{'called':>7}{'valid':>6}{'qual':>6}{'ncall':>6}{'redun':>6}{'match':>6}")
for r in rows:
    print(f"{r['id']:11}{str(r['success']):>5}{str(r['called']):>7}{str(r['valid']):>6}"
          f"{r['qquality']:>6.2f}{r['ncalls']:>6}{r['redund']:>6}{str(r['matches']):>6}")
print("\nreproducible?", rows == rows2)

task        succ called valid  qual ncall redun match
rate-429    True   True  True  1.00     1     0  True
pro-rpm     True   True  True  0.67     1     0  True
pro-cost    True   True  True  0.80     1     0  True
region      True   True  True  0.47     2     0 False
scale-bkup  True   True  True  0.50     1     0  True
sdk-env     True   True  True  1.00     1     0  True
free-cap    True   True  True  1.00     1     0  True
oos-mongo   True   True  True  0.42     3     0 False

reproducible? True

The combined table is the whole evaluation on one screen, and the second score_all returns an identical list — rows == rows2 is True — because scoring never touches the model; it reads the trajectories we froze in Stage 2. Scan the columns and the three levels tell three different stories. The succ, called, and valid columns are solid True all the way down: every task succeeded, every task used the tool, every query was well-formed. But the last three columns break the tie. Look at region and oos-mongo: both match=False, because region took two searches and oos-mongo took three, so neither matched the single-search reference path — even though both got the right answer. The redund column is 0 everywhere, which is the good news: the agent’s extra searches used different queries, so it was exploring, not looping. Notice too that qual varies widely (0.42 to 1.00) without ever causing a failure — a low-overlap query like the refusal probe’s still did its job, which is why query quality is a diagnostic, not a pass/fail gate.

Stage 4: Read the Results#

A table of booleans isn’t a decision. The last step collapses the three levels into an agent scorecard — one number per level — and then lists exactly which tasks to look at. Success rate is the fraction of tasks whose final answer passed. Tool-call correctness collapses Level 2 to a per-task pass (called the tool and valid args and a non-empty query overlap) and averages it. Trajectory quality counts the tasks that took the reference path with no redundant steps. Then we print the weak paths: any task that failed outright, succeeded but didn’t match the reference path, or wasted a repeated search.

n = len(rows)
success_rate = sum(r["success"] for r in rows) / n

def tc_correct(r):                                  # Level-2 collapsed to a per-task pass
    return 1.0 if (r["called"] and r["valid"] and r["qquality"] > 0) else 0.0
tool_correctness = sum(tc_correct(r) for r in rows) / n

traj_quality = sum(1.0 for r in rows if r["matches"] and r["redund"] == 0) / n

print("=== AGENT SCORECARD ===")
print(f"task success rate     : {success_rate:.2f}  ({sum(r['success'] for r in rows)}/{n})")
print(f"tool-call correctness : {tool_correctness:.2f}")
print(f"trajectory quality    : {traj_quality:.2f}")

print("\n--- weak paths (right answer or not, poor path) ---")
for r, run in zip(rows, runs):
    flags = []
    if not r["success"]:                     flags.append("TASK FAILED")
    if r["success"] and not r["matches"]:    flags.append("poor path")
    if r["redund"] > 0:                      flags.append(f"redundant x{r['redund']}")
    if flags:
        surfaced = run["task"]["doc"] in [i for tc in tool_calls(run) for i in tc["result_ids"]]
        print(f"{r['id']:11} {', '.join(flags):18} ncalls={r['ncalls']} matches={r['matches']} relevant_doc_surfaced={surfaced}")
        print(f"            answer: {run['answer'][:88]}")

=== AGENT SCORECARD ===
task success rate     : 1.00  (8/8)
tool-call correctness : 1.00
trajectory quality    : 0.75

--- weak paths (right answer or not, poor path) ---
region      poor path          ncalls=2 matches=False relevant_doc_surfaced=True
            answer: No, you cannot change a database's region after it is created. According to the documenta
oos-mongo   poor path          ncalls=3 matches=False relevant_doc_surfaced=False
            answer: I don't have that in the docs. The search results don't contain information about whether

There is the scorecard, and its shape is the lesson: success 1.00, tool calls 1.00, trajectory 0.75. If you’d only measured outcome, you’d ship this agent with a perfect grade. If you’d only measured tool calls, still perfect. It’s the trajectory level — and only the trajectory level — that docks a quarter of the score and points at the two tasks worth fixing. Both region and oos-mongo got the right answer (that’s why they’re not TASK FAILED) but reached it down a non-reference path: region searched twice, oos-mongo searched three times before its correct refusal. This is the canonical agent finding — succeeded, but took a poor path — and it’s invisible to any final-answer check.

Two subtleties worth reading off. First, the free-cap task — the one that broke Docent in Module 5 — is not on the weak-path list. The agent’s self-authored query surfaced ratelimits on the first try, so it passed all three levels. That’s a real result, but a fragile one: the agent got lucky with its phrasing, and a differently-worded user question could still miss, which is why you keep the retriever-sensitive task in the set and watch it across runs rather than trusting one green light. Second, honesty about variation: because the agent picks its own step count, trajectory quality is the one number that moves run to run — on our re-runs it landed between 0.75 and 0.88 as region sometimes took one search instead of two — while success rate and tool-call correctness stayed pinned at 1.00. The deterministic scoring over any fixed collected set is exact; it’s the collection that varies.

So what would you fix? Not the prompt for correctness — outcomes are perfect. The target is path efficiency on dead-end questions: the out-of-scope probe burning three searches is wasted latency and cost on every unanswerable query, and a real fleet has many. The fix lives in the agent’s stopping policy — a tighter system instruction (“if the first search doesn’t contain the answer, refuse; do not keep searching”) or a hard cap of one search for questions that look out of scope — and the trajectory metric you built is exactly the gauge that tells you whether the fix worked.

Practice Exercises#

Exercise 1: Add a redundancy failure and watch the metric catch it#

Every task in our run scored redund=0 — the agent never repeated a query verbatim. Prove the metric works by constructing a trajectory that does loop: take the collected oos-mongo run and manually duplicate its first tool call in a copy of the trajectory, then re-score just that copy. Confirm redundancy jumps to 1 and that matches_reference is still False, and describe what a nonzero redundancy count would mean about the agent in production.

Exercise 2: Test the stopping-policy fix on the out-of-scope probe#

Stage 4 proposed fixing path efficiency by making the agent give up sooner on dead-end questions. Try it: add one sentence to agentic_docent’s system prompt — something like “If your first search does not contain the answer, refuse immediately; do not search again” — re-run just the oos-mongo task a few times, and check whether num_tool_calls drops toward 1 while the refusal (the correct outcome) survives. Re-score and see whether trajectory quality for that task moves from False to True.

Exercise 3: Weight the scorecard for what your product actually cares about#

Right now the three scorecard numbers stand alone. Imagine two products: a support bot where a wrong answer is a disaster but an extra search is fine, and a latency-critical API where every tool call costs money. Write a single overall(weights) function that combines the three levels into one weighted score, then show how the same Docent run gets a different overall grade under a success-heavy weighting versus a trajectory-heavy one — and argue which weighting you’d defend to a product manager.

Summary#

You evaluated the agentic Docent end to end, at every level this module taught, and produced the artifact a team acts on: an agent scorecard plus a short list of what to fix. You assembled a task set of eight golden questions — seven factual plus the out-of-scope refusal — each paired with a deterministic verifier. You ran the agent live on claude-haiku-4-5 and collected, per task, its final answer and its full recorded trajectory of tool calls. You scored all three levels deterministically over the collected runs — task success, tool-call correctness (called_search, args_valid, query_quality), and trajectory metrics (num_tool_calls, redundancy, matches_reference) — and confirmed the numbers reproduce exactly (rows == rows2 is True). And you read the scorecard: success rate 1.00, tool-call correctness 1.00, trajectory quality 0.75, with the trajectory level alone exposing two tasks that reached the right answer down a wasteful path — the out-of-scope probe burning three searches before its correct refusal.

Key Concepts#

• Agent task set — golden questions turned into tasks by attaching a machine-checkable verify to each; the deterministic verifier is the Level-1 outcome check and everything downstream trusts its boolean.
• Collected run record — the per-task bundle of answer and trajectory, gathered from one live agent pass and then frozen; collecting once is what lets the scoring be reproducible even though the agent isn’t.
• Three-level scoring — task success (right answer?), tool-call correctness (right tool, valid args, sensible query?), and trajectory quality (efficient, non-redundant, reference-shaped path?); you need all three because they can disagree.
• Reference trajectory — the ideal path (one search that surfaces the relevant doc, then answer); matches_reference measures a run against it, catching agents that reach the right answer by wandering.
• Agent scorecard — one number per level plus a weak-path list; its value is that the numbers can diverge, so a perfect outcome grade can sit beside a docked trajectory grade and tell you exactly where to invest.

Why This Matters#

In production an agent’s failure and its inefficiency arrive as the same screen — an answer — and only one of them is visible in that answer. A final-answer check will happily green-light an agent that reaches every correct answer after three redundant tool calls, and you’ll ship escalating latency and cost without a single failing test to warn you. The harness you built here refuses that blind spot: it grades outcome, tool calls, and trajectory as separate numbers, keeps the live randomness in one frozen collection step so the metrics reproduce, and hands you a scorecard where a poor path can’t hide behind a right answer. That separation is what lets you tune an agent deliberately — tighten the stopping policy for latency, or the verifier for correctness — instead of chasing one blended number that tells you something is wrong but never where. Every agent you evaluate from here needs this three-level discipline, because with agents the path is the product as much as the answer is.

Continue Building Your Skills#

You just built the harness an agent team runs before every release: it turns golden questions into scored tasks, runs the agent live once to capture each answer and the path it took, then grades three levels deterministically so a right answer can’t hide a wasteful path. You watched Docent post a perfect outcome and a perfect tool-call score while the trajectory level quietly flagged an out-of-scope question that burned three searches before refusing — the failure mode a final-answer check never sees. Carry the instinct forward: when an agent works, don’t stop at “it answered correctly” — ask how many steps it took, whether it repeated itself, and whether the relevant page ever surfaced, because the path is where cost, latency, and the next regression live. In Module 7, Docent leaves the golden set behind and meets real traffic, and you’ll learn to capture these same trajectories as live traces — so the wandering path shows up on a dashboard, not just in a test.