Lesson 3 - Tool-Call Correctness

Welcome to Tool-Call Correctness#

In Lesson 2 you asked whether agentic Docent – our assistant for the Meridian serverless database docs – accomplished the task. But task success is an outcome metric: it grades the destination and stays silent about the journey. An agent can reach the right answer through a call it should never have made, and it can produce a wrong answer because of a single bad tool call buried three steps back. This lesson zooms all the way in on those calls. We stop looking at the final answer entirely and evaluate the tool calls themselves – one layer of the trajectory, scored on its own terms.

Three questions define a correct tool call, and they are genuinely different questions. First, tool selection: did Docent call the right tool for the job – and just as important, did it call the tool at all when it should have, rather than answering a docs question from the model’s parametric memory? Second, argument validity: were the arguments schema-valid – a query that is actually a non-empty string the way the tool’s input_schema demands? Third, argument quality: a query can be perfectly schema-valid and still be useless ("" parses as… nothing; "please help" parses fine but retrieves nothing relevant). “It parsed” and “it was a good query” are two separate bars, and a serious tool-call evaluation checks both.

In this lesson, you will:

• Run agentic Docent live on a few Meridian tasks and collect each recorded trajectory – the search_docs calls with their query and result_ids
• Write three deterministic checkers over recorded tool calls: called_search (tool selection), args_valid (schema validity), and a query-quality heuristic (does the query share a keyword with the question?)
• Score tool-call correctness across a fixed task list – tool-call accuracy 0.778, precision 0.857, recall 0.857, argument-validity 0.857, query-quality 0.571 – and confirm the checkers run byte-identical twice
• Flag two failure modes on hand-built trajectories: answering from memory (no tool call – a costly false negative) and an empty/garbage query (schema-invalid or low-quality)

The agent under test#

Module 6 evaluates an agentic Docent that decides for itself when to search. It is a standard Anthropic tool-use loop over the Meridian corpus, and crucially it records a trajectory – a list of every search_docs call it makes, with the query it passed and the result_ids that came back. That recording is what makes tool-call evaluation possible: we score the trajectory, not a live model, so the scoring is deterministic and repeatable even though the agent that produced it is not.

import warnings
warnings.filterwarnings("ignore")
import anthropic, json

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

DOCS = [
    {"id": "auth", "title": "Authentication",
     "text": "Meridian authenticates every request with an API key sent in the Authorization header as a bearer token. "
             "Create keys in the dashboard under Settings > API Keys. Keys are either test or live mode; test keys only "
             "touch sandbox data. A key is shown once at creation and cannot be recovered, only rotated."},
    {"id": "ratelimits", "title": "Rate Limits",
     "text": "The Free plan allows 60 requests per minute. The Pro plan allows 600 requests per minute. Exceeding the "
             "limit returns HTTP 429 with a Retry-After header giving the seconds to wait. Rate limits are applied per "
             "API key, not per account."},
    {"id": "plans", "title": "Plans and Pricing",
     "text": "Meridian has three plans. Free costs 0 dollars per month and includes 1 GB of storage. Pro costs 25 dollars "
             "per month and includes 50 GB of storage. Scale costs 199 dollars per month and includes 500 GB of storage. "
             "All plans include unlimited read replicas."},
    {"id": "regions", "title": "Regions",
     "text": "Databases can be created in three regions: us-east, eu-west, and ap-south. The region is fixed at creation "
             "time and cannot be changed later; to move data to another region you must export and re-import into a new "
             "database. Cross-region read replicas are available on the Scale plan only."},
    {"id": "errors", "title": "Error Codes",
     "text": "Meridian uses standard HTTP status codes. 400 means a malformed request, 401 means a missing or invalid API "
             "key, 403 means the key is valid but lacks permission, 404 means the resource does not exist, and 429 means "
             "the rate limit was exceeded. 5xx codes indicate a server-side error and should be retried with backoff."},
    {"id": "backups", "title": "Backups",
     "text": "Automatic daily backups are retained for 7 days on the Free plan, 30 days on Pro, and 90 days on Scale. "
             "Backups run at 03:00 UTC. Point-in-time recovery to any second in the retention window is available on the "
             "Scale plan. Restoring a backup creates a new database and never overwrites the original."},
    {"id": "sdks", "title": "SDKs",
     "text": "Official SDKs are available for Python, JavaScript, and Go. Install the Python SDK with 'pip install meridian'. "
             "The SDK reads the MERIDIAN_API_KEY environment variable by default. All SDKs retry idempotent requests up to "
             "3 times on 5xx errors with exponential backoff."},
    {"id": "deletion", "title": "Deleting Data",
     "text": "Deleting a database is immediate and irreversible once the 24-hour grace period ends. During the grace period "
             "the database is soft-deleted and can be restored from the dashboard. After 24 hours the data and its backups "
             "are permanently purged. Deletion requires a live-mode key with admin permission."},
]

SEARCH_TOOL = [{
    "name": "search_docs",
    "description": "Search the Meridian documentation and return matching pages. Call this before answering.",
    "input_schema": {
        "type": "object",
        "properties": {"query": {"type": "string", "description": "search query"}},
        "required": ["query"],
    },
}]

def search_docs(query, docs, k=2):
    q = set(query.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())), reverse=True)
    hits = scored[:k]
    return [{"id": d["id"], "title": d["title"], "text": d["text"]} for d in hits]

def agentic_docent(question, docs, model="claude-haiku-4-5", max_steps=4):
    """Returns (final_answer, trajectory). trajectory records each step:
       {"type": "tool_call", "tool": "search_docs", "input": {...}, "result_ids": [...]} or
       {"type": "final", "text": ...}."""
    system = ("You are Docent, a Meridian docs assistant. Use the search_docs tool to find relevant pages, "
              "then answer ONLY from what it returns. If the docs don't contain the answer, say "
              "\"I don't have that in the docs.\"")
    messages = [{"role": "user", "content": question}]
    trajectory = []
    for _ in range(max_steps):
        resp = client.messages.create(model=model, max_tokens=400, system=system,
                                       tools=SEARCH_TOOL, messages=messages)
        if resp.stop_reason == "tool_use":
            messages.append({"role": "assistant", "content": resp.content})
            tool_results = []
            for block in resp.content:
                if block.type == "tool_use":
                    hits = search_docs(block.input["query"], docs)
                    trajectory.append({"type": "tool_call", "tool": block.name,
                                       "input": block.input, "result_ids": [h["id"] for h in hits]})
                    tool_results.append({"type": "tool_result", "tool_use_id": block.id,
                                         "content": json.dumps(hits)})
            messages.append({"role": "user", "content": tool_results})
        else:
            text = "".join(b.text for b in resp.content if b.type == "text")
            trajectory.append({"type": "final", "text": text})
            return text, trajectory
    return "(stopped: max steps)", trajectory

The key line for us is where the trajectory records {"type": "tool_call", "input": block.input, "result_ids": [...]}. That is the raw material of tool-call evaluation: after a run we have, for every step, exactly which tool was called and with what arguments. Everything below reads that structure and never calls the model again.

Collecting real trajectories#

Let’s run agentic Docent on five tasks drawn from the module’s golden set – four in-scope Meridian questions plus the out-of-scope MongoDB probe, whose correct behavior is to refuse. We keep the calls small (short questions, max_tokens=400) and print, for each task, how many tool calls it made, the first query string, and the returned result_ids.

TASKS = [
    "What HTTP status code does Meridian return when you exceed the rate limit?",
    "How much does the Pro plan cost per month?",
    "What does a 401 error mean?",
    "Which environment variable does the Python SDK read for the API key?",
    "Does Meridian support MongoDB-style aggregation pipelines?",   # out of scope
]

for q in TASKS:
    answer, traj = agentic_docent(q, DOCS)
    calls = [s for s in traj if s["type"] == "tool_call"]
    qtext = calls[0]["input"].get("query") if calls else None
    print(f"\nQ: {q}")
    print(f"   tool calls: {len(calls)}  first query: {qtext!r}")
    print(f"   result_ids: {[c['result_ids'] for c in calls]}")
    print(f"   answer: {answer[:90]!r}")

Q: What HTTP status code does Meridian return when you exceed the rate limit?
   tool calls: 1  first query: 'rate limit HTTP status code'
   result_ids: [['errors', 'ratelimits']]
   answer: 'When you exceed the rate limit in Meridian, the API returns **HTTP status code 429**.\n\nAdd'

Q: How much does the Pro plan cost per month?
   tool calls: 1  first query: 'Pro plan cost price per month'
   result_ids: [['ratelimits', 'plans']]
   answer: 'The **Pro plan costs $25 per month** and includes 50 GB of storage.'

Q: What does a 401 error mean?
   tool calls: 1  first query: '401 error'
   result_ids: [['errors', 'auth']]
   answer: 'According to the Meridian documentation, a **401 error means a missing or invalid API key*'

Q: Which environment variable does the Python SDK read for the API key?
   tool calls: 1  first query: 'Python SDK environment variable API key'
   result_ids: [['sdks', 'auth']]
   answer: 'The Python SDK reads the **`MERIDIAN_API_KEY`** environment variable by default for the AP'

Q: Does Meridian support MongoDB-style aggregation pipelines?
   tool calls: 3  first query: 'MongoDB aggregation pipeline'
   result_ids: [['auth', 'ratelimits'], ['auth', 'ratelimits'], ['auth', 'plans']]
   answer: "I don't have that in the docs. The documentation I have access to covers authentication, r"

This is a real run, and because the model is live it will not be byte-identical if you run it again – the exact query strings and the number of retries on the out-of-scope question will drift. What is stable is the shape: on the four in-scope questions Docent called search_docs exactly once with a query that clearly targets the right page, then answered from what came back. On the MongoDB question it searched several times (the keyword retriever keeps returning irrelevant pages because no page mentions MongoDB) before correctly giving up. Eyeballing five trajectories is fine for a demo; to score hundreds we need checkers that read the recorded calls mechanically.

Three deterministic checkers#

Because we score recorded calls, no model is involved and the checks are pure functions. Each targets one of the three questions from the intro.

called_search answers tool selection at its most basic: did this trajectory call search_docs at all? For a Meridian docs question the expected behavior is yes; a trajectory with zero calls means Docent answered from parametric memory, which is the failure we most want to catch.

args_valid answers schema validity: the tool’s input_schema requires query to be a string, and a search with an empty or whitespace-only query is degenerate. This is exactly the check a JSON-schema validator would do, written out so you can see it.

query_quality answers argument quality, which no schema can express: does the query share at least one content keyword with the question? It is a deliberately simple heuristic – strip short and stop words, and intersect – but it separates a targeted query like "rate limit HTTP status code" from a useless one like "please help" that parses fine yet retrieves nothing on point.

STOP = {"what", "does", "the", "a", "an", "is", "are", "how", "much", "long",
        "for", "when", "you", "do", "did", "of", "to", "and", "that", "this",
        "there", "everything", "thats", "read"}

def content_words(text):
    toks = "".join(c.lower() if c.isalnum() or c.isspace() else " " for c in text).split()
    return {t for t in toks if len(t) >= 3 and t not in STOP}

def called_search(calls):
    """Tool selection: did the trajectory call search_docs at all?"""
    return any(c["tool"] == "search_docs" for c in calls)

def args_valid(call):
    """Schema check: 'query' must be a present, non-empty string per the input_schema."""
    q = call.get("input", {}).get("query")
    return isinstance(q, str) and q.strip() != ""

def query_quality(question, call):
    """Semantic check: does the query share a content keyword with the question?"""
    if not args_valid(call):
        return False
    return len(content_words(call["input"]["query"]) & content_words(question)) > 0

Note that query_quality returns False for an invalid query without crashing – a call that fails the schema check cannot pass the quality check, so the two form a ladder: parse first, then judge usefulness.

Scoring a fixed batch of recorded calls#

To get stable, reproducible numbers we freeze a list of recorded trajectories – the kind agentic_docent produces, but pinned so the arithmetic is deterministic. Each entry carries the question, whether it should have searched (should_search), and the recorded calls. Four are clean in-scope runs; then we include three deliberately broken ones and two chit-chat turns that need no search at all – so the batch exercises every checker and both directions of the tool-selection decision.

RECORDED = [
    # in-scope tasks handled well: one search, a sensible query
    {"q": "What HTTP status code does Meridian return when you exceed the rate limit?",
     "should_search": True,
     "calls": [{"tool": "search_docs", "input": {"query": "rate limit HTTP status code"},
                "result_ids": ["errors", "ratelimits"]}]},
    {"q": "How much does the Pro plan cost per month?",
     "should_search": True,
     "calls": [{"tool": "search_docs", "input": {"query": "Pro plan cost price per month"},
                "result_ids": ["ratelimits", "plans"]}]},
    {"q": "What does a 401 error mean?",
     "should_search": True,
     "calls": [{"tool": "search_docs", "input": {"query": "401 error"},
                "result_ids": ["errors", "auth"]}]},
    {"q": "Which environment variable does the Python SDK read for the API key?",
     "should_search": True,
     "calls": [{"tool": "search_docs", "input": {"query": "Python SDK environment variable API key"},
                "result_ids": ["sdks", "auth"]}]},
    # NEGATIVE 1: answered from parametric memory, never searched (tool-selection miss)
    {"q": "How long is the grace period before a deleted database is permanently purged?",
     "should_search": True, "calls": []},
    # NEGATIVE 2: searched, but the query is EMPTY (schema-invalid argument)
    {"q": "How long are backups retained on the Scale plan?",
     "should_search": True,
     "calls": [{"tool": "search_docs", "input": {"query": ""},
                "result_ids": ["auth", "ratelimits"]}]},
    # NEGATIVE 3: searched with a non-empty but USELESS query (valid, poor quality)
    {"q": "Which plans support point-in-time recovery?",
     "should_search": True,
     "calls": [{"tool": "search_docs", "input": {"query": "please help"},
                "result_ids": ["auth", "plans"]}]},
    # chit-chat needing NO search; agent correctly stayed quiet (true negative)
    {"q": "Thanks, that's everything!", "should_search": False, "calls": []},
    # chit-chat, but the agent searched anyway: an unnecessary call (precision hit)
    {"q": "Hi there!", "should_search": False,
     "calls": [{"tool": "search_docs", "input": {"query": "hi there"},
                "result_ids": ["auth", "ratelimits"]}]},
]

def score():
    tp = fp = fn = tn = 0
    all_calls = valid_calls = quality_calls = 0
    rows = []
    for rec in RECORDED:
        did, should = called_search(rec["calls"]), rec["should_search"]
        if should and did:       tp += 1
        elif should and not did:  fn += 1
        elif not should and did:  fp += 1
        else:                     tn += 1
        first = rec["calls"][0] if rec["calls"] else None
        av = args_valid(first) if first else None
        qq = query_quality(rec["q"], first) if first else None
        for c in rec["calls"]:
            all_calls += 1
            valid_calls += args_valid(c)
            quality_calls += query_quality(rec["q"], c)
        rows.append((rec["q"], should, did, av, qq))

    print(f"{'search?':>8}{'should':>8}{'valid':>7}{'quality':>9}   question")
    print("-" * 78)
    for q, should, did, av, qq in rows:
        m = lambda x: "-" if x is None else ("ok" if x else "FAIL")
        ok = (did == should) and (av in (None, True)) and (qq in (None, True))
        print(f"{m(did):>8}{str(should):>8}{m(av):>7}{m(qq):>9}   {q[:36]}{'' if ok else '  <-- flagged'}")

    n = len(RECORDED)
    acc = (tp + tn) / n
    precision = tp / (tp + fp) if (tp + fp) else 0.0
    recall = tp / (tp + fn) if (tp + fn) else 0.0
    print()
    print(f"tool-call accuracy (task)  = {acc:.3f}   (TP={tp} TN={tn} FP={fp} FN={fn})")
    print(f"tool-selection precision   = {precision:.3f}")
    print(f"tool-selection recall      = {recall:.3f}")
    print(f"argument-validity rate     = {valid_calls/all_calls:.3f}   ({valid_calls}/{all_calls} calls)")
    print(f"query-quality rate         = {quality_calls/all_calls:.3f}   ({quality_calls}/{all_calls} calls)")
    return (round(acc, 3), round(precision, 3), round(recall, 3),
            round(valid_calls/all_calls, 3), round(quality_calls/all_calls, 3))

result = score()

 search?  should  valid  quality   question
------------------------------------------------------------------------------
      ok    True     ok       ok   What HTTP status code does Meridian
      ok    True     ok       ok   How much does the Pro plan cost per
      ok    True     ok       ok   What does a 401 error mean?
      ok    True     ok       ok   Which environment variable does the
    FAIL    True      -        -   How long is the grace period before   <-- flagged
      ok    True   FAIL     FAIL   How long are backups retained on th   <-- flagged
      ok    True     ok     FAIL   Which plans support point-in-time r   <-- flagged
    FAIL   False      -        -   Thanks, that's everything!
      ok   False     ok     FAIL   Hi there!  <-- flagged

tool-call accuracy (task)  = 0.778   (TP=6 TN=1 FP=1 FN=1)
tool-selection precision   = 0.857
tool-selection recall      = 0.857
argument-validity rate     = 0.857   (6/7 calls)
query-quality rate         = 0.571   (4/7 calls)

Read the table top to bottom and each metric earns its number. Tool-call accuracy = 0.778: seven of nine tasks made the right decision about whether to search (six should-and-did, one shouldn’t-and-didn’t), while two got it wrong. Those two wrong decisions are the interesting ones. The grace-period task should have searched and didn’t – a false negative, Docent answering a docs question from memory. The "Hi there!" turn shouldn’t have searched but did – a false positive, an unnecessary call. That split is exactly what precision and recall separate: precision = 0.857 penalizes the needless call, recall = 0.857 penalizes the missed one.

The argument checks tell a second, independent story about the seven calls that were made. Argument-validity = 0.857 because six of seven queries are non-empty strings; the one empty-query call fails the schema outright. Query-quality = 0.571 is the harshest number here, and deliberately so: only four of seven queries actually share a keyword with their question. The empty query fails (nothing to match), "please help" fails (valid but off-topic), and "hi there" fails (a search that shouldn’t exist can’t be on-topic). A tool call can clear the schema bar and still be a bad call – that gap between 0.857 valid and 0.571 good is the whole point of checking quality on top of validity.

Determinism: the same recorded calls score the same twice#

The live agent varies run to run, but the checkers are pure functions of a fixed input. Run score() a second time on the identical RECORDED list and every number matches to the digit – that reproducibility is what lets you track tool-call correctness across agent versions and trust that any movement is real signal, not sampling noise.

result2 = score()
print("\nidentical:", result == result2)
print("metrics:", result)

 search?  should  valid  quality   question
------------------------------------------------------------------------------
      ok    True     ok       ok   What HTTP status code does Meridian
      ok    True     ok       ok   How much does the Pro plan cost per
      ok    True     ok       ok   What does a 401 error mean?
      ok    True     ok       ok   Which environment variable does the
    FAIL    True      -        -   How long is the grace period before   <-- flagged
      ok    True   FAIL     FAIL   How long are backups retained on th   <-- flagged
      ok    True     ok     FAIL   Which plans support point-in-time r   <-- flagged
    FAIL   False      -        -   Thanks, that's everything!
      ok   False     ok     FAIL   Hi there!  <-- flagged

tool-call accuracy (task)  = 0.778   (TP=6 TN=1 FP=1 FN=1)
tool-selection precision   = 0.857
tool-selection recall      = 0.857
argument-validity rate     = 0.857   (6/7 calls)
query-quality rate         = 0.571   (4/7 calls)

identical: True
metrics: (0.778, 0.857, 0.857, 0.857, 0.571)

Byte-identical, as a deterministic checker must be. Collecting trajectories is live and noisy; scoring them is fixed and reproducible – keep those two phases separate and you get the best of both.

The two failures that matter most#

Two rows in that table deserve to be looked at on their own, because they are the tool-call failures you will actually chase in production. Here they are as hand-built trajectories – the exact shape agentic_docent records – run through the same three checkers.

memory_answer = {   # Docent went straight to a final answer, no search_docs call
    "q": "How long is the grace period before a deleted database is permanently purged?",
    "calls": []}
garbage_query = {   # Docent called the tool, but with an empty query
    "q": "How long are backups retained on the Scale plan?",
    "calls": [{"tool": "search_docs", "input": {"query": ""}, "result_ids": ["auth", "ratelimits"]}]}

for label, rec in [("answered-from-memory", memory_answer), ("empty-query", garbage_query)]:
    did = called_search(rec["calls"])
    first = rec["calls"][0] if rec["calls"] else None
    print(f"[{label}]  called_search={did}  "
          f"args_valid={args_valid(first) if first else None}  "
          f"query_quality={query_quality(rec['q'], first) if first else None}")
    if not did:
        print("   -> TOOL-SELECTION FAILURE: answered without searching the docs")
    elif not args_valid(first):
        print("   -> ARGUMENT FAILURE: empty query violates the tool schema")

[answered-from-memory]  called_search=False  args_valid=None  query_quality=None
   -> TOOL-SELECTION FAILURE: answered without searching the docs
[empty-query]  called_search=True  args_valid=False  query_quality=False
   -> ARGUMENT FAILURE: empty query violates the tool schema

The answered-from-memory trajectory is the dangerous one. called_search is False, so the argument checks never even apply – there is no call to inspect. Docent produced a fluent answer straight from the model’s parametric memory, which for a docs assistant is a correctness landmine: the model might recall Meridian’s grace period as 24 hours, or it might confidently invent 48. A final-answer check that happens to match would call this a pass; the tool-call check catches it as a missed call regardless of whether the guess was lucky. “Did not call the tool when it should have” is the most expensive tool-selection failure precisely because it hides behind a plausible answer.

The empty-query trajectory fails one gate earlier than you might expect. It did call the tool, so tool selection passes – but args_valid is False, and since an invalid call cannot be a quality call, query_quality is False too. This is where separating the two checks pays off: the agent tried to do the right thing (search) but botched the argument, a different bug with a different fix than not searching at all.

Practice Exercises#

Exercise 1: Catch a wrong-tool call, not just a missing one#

called_search only checks whether search_docs was called. Imagine Docent gains a second tool, create_ticket, that it must never call for a read-only docs question. Write a checker only_expected_tools(calls, allowed={"search_docs"}) that returns False if the trajectory calls any tool outside allowed, and test it on a hand-built trajectory whose calls include one search_docs call and one create_ticket call. Explain why “called the right tool” and “called only the right tools” are different checks.

Exercise 2: Precision and recall of the actual calls, not the task decision#

The lesson scored tool selection at the task level (did this task search, yes or no). Instead, score it at the call level against an expected call set. For each recorded task add an expected count (in-scope = 1 search, chit-chat = 0), then across the batch compute call-level precision = (matched calls) / (calls made) and recall = (matched calls) / (calls expected). Compare your numbers to the task-level 0.857 / 0.857 and explain where the out-of-scope MongoDB trajectory, which searched three times, would push each metric.

Exercise 3: Stress-test the query-quality heuristic#

query_quality is a keyword-overlap heuristic, so it has blind spots. Find a query that is genuinely good but that the heuristic marks False (a false negative of the checker), and a query that is genuinely useless but marked True (a false positive). Use questions from the golden set. Then describe one concrete improvement – and one new failure it would introduce.

Summary#

Tool-call correctness evaluates an agent’s tool calls independently of its final answer, because the two fail in different ways and one metric cannot see both. You ran agentic Docent live on five Meridian tasks, collected each recorded trajectory (the search_docs calls with their query and result_ids), then applied three deterministic checkers: called_search for tool selection, args_valid for schema validity, and a keyword-overlap query_quality heuristic for argument quality. Scored over a fixed nine-task batch, the checkers reported tool-call accuracy 0.778 (TP=6, TN=1, FP=1, FN=1), tool-selection precision and recall both 0.857, an argument-validity rate of 0.857 (6/7 calls schema-valid), and a query-quality rate of 0.571 (only 4/7 queries actually useful) – and re-running produced byte-identical numbers, because scoring recorded calls is deterministic even though collecting them is not. Two negative trajectories isolated the failures that matter most: answering from parametric memory with no search_docs call at all – a false negative that hides behind a plausible answer and is the costliest tool-selection error – and calling the tool with an empty query, which passes tool selection but fails the schema and quality gates. The recurring lesson: schema-valid and genuinely useful are two separate bars, and a serious tool-call evaluation checks both.

Key Concepts#

• Tool selection – did the agent call the right tool, and call it at all when it should? Checked by called_search; the failure to catch is answering from memory.
• Argument validity – are the tool arguments schema-valid (here a non-empty query string per the tool’s input_schema)? Checked by args_valid; Docent: 0.857 of calls.
• Argument quality – is the argument actually good, not just parseable? A "" or "please help" query is valid-or-degenerate yet useless. Checked by query_quality; Docent: 0.571.
• Tool-call accuracy – fraction of tasks that made the right search-or-not decision; 0.778 here (TP=6, TN=1, FP=1, FN=1).
• Precision vs recall of tool calls – precision (0.857) penalizes calls the agent shouldn’t have made; recall (0.857) penalizes calls it should have made and missed. A missed call is a false negative; a needless call is a false positive.
• Determinism – collecting trajectories is live and noisy; scoring them with pure-function checkers is fixed and reproducible. Keep the two phases separate.

Why This Matters#

When an agentic Docent answer goes wrong, task success tells you that it failed and tool-call correctness tells you where. A low recall means Docent is skipping the search and answering from memory – a factual landmine no answer-only check reliably catches, because the guess is sometimes right. A low argument-validity rate means malformed calls that waste steps and can crash a strict tool runtime. And the gap between a high validity rate and a lower quality rate – 0.857 versus 0.571 here – is the difference between “the call parsed” and “the call was worth making,” the single most common blind spot in agent evaluation. Scoring tool calls on their own turns a vague “the agent seems flaky” into a precise diagnosis: wrong tool, missing call, invalid argument, or valid-but-useless query – each with a different fix.

Continue Building Your Skills#

You can now grade an agent’s tool calls without ever looking at its final answer: three deterministic gates – right tool, valid arguments, good query – turned five live Docent trajectories and a fixed nine-task batch into hard numbers (accuracy 0.778, precision and recall 0.857, validity 0.857, quality 0.571), with two negative examples pinning down the costliest failure, answering from memory with no search at all. But a single good call is not a good path. Docent might call search_docs correctly and then call it four more times, or search, half-answer, search again, and loop until it runs out of steps – every individual call valid, the trajectory as a whole a mess. The next lesson evaluates that whole sequence: trajectory evaluation, where you measure whether the path from question to answer was efficient and sound, not just whether each step in it was well formed.