Lesson 2 - Task Success & Outcome Metrics

Welcome to Task Success & Outcome Metrics#

In Lesson 1 you watched a single answer unfold into a whole path: the agentic Docent – our assistant for the Meridian serverless database docs – decides for itself to call search_docs, reads what comes back, and only then answers. That path is rich to inspect, and the next two lessons dig into it. But before we grade how the agent worked, we have to answer the blunt question a user actually cares about: did it get the job done? For Docent, “the job” is a correct, grounded answer to a docs question – and, just as importantly, a clean refusal when the docs can’t answer at all.

This lesson is about the outcome side of agent evaluation and its headline metric, task success rate: the fraction of tasks the agent completed correctly. It sounds simple, and that simplicity is exactly why it’s the first number anyone looks at. The catch is that “correctly” has to be made checkable: every task needs a verifier – a rule, a function, or a judge that looks at the end result and returns pass or fail. We’ll build one for Docent, run it live over eight golden tasks, and get a real success rate. Then, because a metric you never see fail is a metric you can’t trust, we’ll break the agent’s retriever on purpose and confirm the number reacts.

In this lesson, you will:

• Define task success rate and why it is the headline metric for any agent
• Write a task_succeeded verifier that checks Docent’s final answer against a reference fact, and checks the out-of-scope task for a correct refusal
• Run the agentic Docent live over 8 golden tasks and compute a real per-task pass/fail board (8/8 on this run)
• Weigh binary success vs partial credit, and distinguish exact end-state checks, verifier functions, and LLM judges
• Inject a broken retriever into the same agent and watch task success collapse to 1/8 – the outcome metric catching a regression the eval code never changed

What “task success” means for an agent#

A non-agentic metric asks “is this answer good?” An outcome metric for an agent asks something stricter and more useful: “did the agent achieve the goal it was given?” The unit of measurement is the task, not the token or the sentence – and a task either succeeded or it didn’t. Average that over a suite and you get the task success rate:

Everything hard about this metric lives in the phrase “completed correctly.” You cannot compute a success rate until you have decided, per task, what counts as success – a success criterion that some verifier can check automatically. For a docs assistant like Docent the criterion is concrete:

• For a factual task, success = the final answer contains the correct fact from the Meridian docs (e.g. the answer to “what status code for a rate limit?” contains 429).
• For the out-of-scope task, success = the agent correctly refuses – it says it doesn’t have that in the docs rather than inventing an answer.

Notice both criteria look only at the final answer, not at how many times the agent searched or which query it used. That is deliberate: outcome and process are different axes, and this lesson owns the outcome axis.

The agentic Docent and the golden task set#

We reuse the module’s tool-using Docent verbatim: an Anthropic tool-use loop over the 8-page Meridian corpus, where the model calls a search_docs tool and then answers from what it returns. The one addition here is that agentic_docent takes a swappable retriever argument – everything runs on the real keyword retriever now, but that seam lets us inject a broken retriever later without touching the agent’s logic.

import warnings; warnings.filterwarnings("ignore")
import anthropic, json
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

DOCS = [
    {"id": "auth", "title": "Authentication",
     "text": "Meridian authenticates every request with an API key sent in the Authorization header as a bearer token. "
             "Create keys in the dashboard under Settings > API Keys. Keys are either test or live mode; test keys only "
             "touch sandbox data. A key is shown once at creation and cannot be recovered, only rotated."},
    {"id": "ratelimits", "title": "Rate Limits",
     "text": "The Free plan allows 60 requests per minute. The Pro plan allows 600 requests per minute. Exceeding the "
             "limit returns HTTP 429 with a Retry-After header giving the seconds to wait. Rate limits are applied per "
             "API key, not per account."},
    {"id": "plans", "title": "Plans and Pricing",
     "text": "Meridian has three plans. Free costs 0 dollars per month and includes 1 GB of storage. Pro costs 25 dollars "
             "per month and includes 50 GB of storage. Scale costs 199 dollars per month and includes 500 GB of storage. "
             "All plans include unlimited read replicas."},
    {"id": "regions", "title": "Regions",
     "text": "Databases can be created in three regions: us-east, eu-west, and ap-south. The region is fixed at creation "
             "time and cannot be changed later; to move data to another region you must export and re-import into a new "
             "database. Cross-region read replicas are available on the Scale plan only."},
    {"id": "errors", "title": "Error Codes",
     "text": "Meridian uses standard HTTP status codes. 400 means a malformed request, 401 means a missing or invalid API "
             "key, 403 means the key is valid but lacks permission, 404 means the resource does not exist, and 429 means "
             "the rate limit was exceeded. 5xx codes indicate a server-side error and should be retried with backoff."},
    {"id": "backups", "title": "Backups",
     "text": "Automatic daily backups are retained for 7 days on the Free plan, 30 days on Pro, and 90 days on Scale. "
             "Backups run at 03:00 UTC. Point-in-time recovery to any second in the retention window is available on the "
             "Scale plan. Restoring a backup creates a new database and never overwrites the original."},
    {"id": "sdks", "title": "SDKs",
     "text": "Official SDKs are available for Python, JavaScript, and Go. Install the Python SDK with 'pip install meridian'. "
             "The SDK reads the MERIDIAN_API_KEY environment variable by default. All SDKs retry idempotent requests up to "
             "3 times on 5xx errors with exponential backoff."},
    {"id": "deletion", "title": "Deleting Data",
     "text": "Deleting a database is immediate and irreversible once the 24-hour grace period ends. During the grace period "
             "the database is soft-deleted and can be restored from the dashboard. After 24 hours the data and its backups "
             "are permanently purged. Deletion requires a live-mode key with admin permission."},
]

SEARCH_TOOL = [{
    "name": "search_docs",
    "description": "Search the Meridian documentation and return matching pages. Call this before answering.",
    "input_schema": {"type": "object",
        "properties": {"query": {"type": "string", "description": "search query"}},
        "required": ["query"]},
}]

def search_docs(query, docs, k=2):
    q = set(query.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())), reverse=True)
    return [{"id": d["id"], "title": d["title"], "text": d["text"]} for d in scored[:k]]

def agentic_docent(question, docs, retriever=search_docs, model="claude-haiku-4-5", max_steps=4):
    """Runs the Docent agent. `retriever` is swappable so we can inject a broken one later."""
    system = ("You are Docent, a Meridian docs assistant. Use the search_docs tool to find relevant pages, "
              "then answer ONLY from what it returns. If the docs don't contain the answer, say "
              "\"I don't have that in the docs.\"")
    messages = [{"role": "user", "content": question}]
    trajectory = []
    for _ in range(max_steps):
        resp = client.messages.create(model=model, max_tokens=400, system=system,
                                       tools=SEARCH_TOOL, messages=messages)
        if resp.stop_reason == "tool_use":
            messages.append({"role": "assistant", "content": resp.content})
            tool_results = []
            for block in resp.content:
                if block.type == "tool_use":
                    hits = retriever(block.input["query"], docs)
                    trajectory.append({"type": "tool_call", "result_ids": [h["id"] for h in hits]})
                    tool_results.append({"type": "tool_result", "tool_use_id": block.id,
                                         "content": json.dumps(hits)})
            messages.append({"role": "user", "content": tool_results})
        else:
            text = "".join(b.text for b in resp.content if b.type == "text")
            trajectory.append({"type": "final", "text": text})
            return text, trajectory
    return "(stopped: max steps)", trajectory

The golden task set is the outcome analogue of the golden Q&A you built in Module 3. Each task carries a question, an accept list of strings that a correct answer must contain, and an out_of_scope flag. Seven are factual; the eighth is the MongoDB-aggregation probe, which the docs cannot answer – its success criterion is a refusal.

TASKS = [
    {"q": "What HTTP status code does Meridian return when you exceed the rate limit?",
     "accept": ["429"], "out_of_scope": False},
    {"q": "How much does the Pro plan cost per month?",
     "accept": ["25"], "out_of_scope": False},
    {"q": "Can you change a database's region after it is created?",
     "accept": ["cannot", "can't", "fixed", "export"], "out_of_scope": False},
    {"q": "How long are backups retained on the Scale plan?",
     "accept": ["90"], "out_of_scope": False},
    {"q": "Which environment variable does the Python SDK read for the API key?",
     "accept": ["MERIDIAN_API_KEY"], "out_of_scope": False},
    {"q": "What does a 401 error mean?",
     "accept": ["invalid", "missing"], "out_of_scope": False},
    {"q": "How long is the grace period before a deleted database is purged?",
     "accept": ["24"], "out_of_scope": False},
    {"q": "Does Meridian support MongoDB-style aggregation pipelines?",
     "accept": [], "out_of_scope": True},  # correct behavior: refuse
]

Defining the success check#

The verifier is where an outcome metric lives or dies, so it deserves its own thought. There are three broad ways to check whether a task succeeded, in ascending order of flexibility and cost:

1. Exact end-state check. For an agent that changes the world – writes a file, books a slot, updates a row – you don’t read the transcript at all. You inspect the world afterward: does the file exist with the right contents? Is the slot booked? This is the gold standard when it applies, because it can’t be fooled by an agent that claims success without achieving it.
2. Verifier function. A small deterministic rule over the final answer – keyword or regex match, a numeric comparison, a JSON-schema check. Cheap, reproducible, and perfect when the correct answer has a checkable signature (a status code, an env-var name, a dollar figure).
3. LLM judge. For open-ended tasks where no keyword captures “correct” – a summary, an explanation, a multi-sentence recommendation – you hand the answer and the reference to a judge model and ask it to rule. Flexible, but non-deterministic and needs the calibration you built in Module 4.

Docent is read-only – it never changes state, it just answers – so an exact end-state check doesn’t apply here (there’s no world to inspect). Its answers are short and factual, so we use option 2: a verifier function over the final answer, exactly the kind of keyword/refusal check you’d reuse from a reference-based eval. A factual task passes when the reference fact appears in the answer; the out-of-scope task passes when the agent refuses. One guard matters: a factual task that wrongly refuses – gives up on a fact that was retrievable – must count as a failure, so we detect refusals explicitly and fail a factual task that produced one.

def is_refusal(answer):
    a = answer.lower()
    negation = any(p in a for p in ["don't have", "do not have", "doesn't contain", "does not contain",
                                    "not in the doc", "isn't in the doc", "no information",
                                    "couldn't find", "could not find", "unable to"])
    scope = ("doc" in a) or ("information" in a) or ("search" in a)
    return negation and scope

def task_succeeded(answer, task):
    """Outcome verifier: did the agent accomplish THIS task? Checks the final answer only."""
    if task["out_of_scope"]:
        return is_refusal(answer)                 # success = correctly refusing
    if is_refusal(answer):                        # a factual task the agent wrongly gave up on
        return False
    return any(kw.lower() in answer.lower() for kw in task["accept"])

Running the harness: a real task success rate#

Now wire it together. run_suite runs the agent on each task, applies the verifier, and returns a row per task; report prints the pass/fail board and the aggregate rate. This is a live run – eight real calls to claude-haiku-4-5 through the tool-use loop – so treat the exact board as one honest sample, not a fixed constant.

def run_suite(tasks, retriever=search_docs):
    rows = []
    for t in tasks:
        answer, _ = agentic_docent(t["q"], DOCS, retriever=retriever)
        rows.append({"q": t["q"], "kind": "refuse" if t["out_of_scope"] else "fact",
                     "ok": task_succeeded(answer, t), "answer": " ".join(answer.split())})
    return rows

def report(rows, label):
    print(f"== {label} ==")
    print(f"{'task':<58}{'type':>8}   result")
    print("-" * 82)
    for r in rows:
        print(f"{r['q'][:56]:<58}{r['kind']:>8}   {'PASS' if r['ok'] else 'FAIL'}")
    p = sum(r["ok"] for r in rows)
    print("-" * 82)
    print(f"Task success rate = {p}/{len(rows)} = {p/len(rows):.3f}\n")
    return p

rows = run_suite(TASKS)
report(rows, "Docent agent, real run")

== Docent agent, real run ==
task                                                          type   result
----------------------------------------------------------------------------------
What HTTP status code does Meridian return when you exce      fact   PASS
How much does the Pro plan cost per month?                    fact   PASS
Can you change a database's region after it is created?       fact   PASS
How long are backups retained on the Scale plan?              fact   PASS
Which environment variable does the Python SDK read for       fact   PASS
What does a 401 error mean?                                   fact   PASS
How long is the grace period before a deleted database i      fact   PASS
Does Meridian support MongoDB-style aggregation pipeline    refuse   PASS
----------------------------------------------------------------------------------
Task success rate = 8/8 = 1.000

A clean 8/8 = 1.000 on this run. Seven factual questions got grounded answers containing the reference fact, and the out-of-scope MongoDB question was correctly refused. That’s a genuinely strong result, and it’s worth understanding why: claude-haiku-4-5 rewrites the search query well (so the keyword retriever surfaces the right page even for the tricky “Free plan” style phrasing that stumped the raw retriever back in Module 5), and it refuses out-of-scope questions reliably.

Two honest caveats keep this number from being oversold. First, the run is non-deterministic: re-run it and you’ll usually see 8/8 again, but an occasional out-of-scope phrasing or a borderline answer can flip a single task, so the rate may read 7/8 on some runs. That variance is itself a reason to evaluate over a set and to re-run, rather than trusting one question one time. Second – and more importantly – a metric that only ever prints “everything passed” is a metric you have no reason to trust. How do you know task_succeeded would even notice a broken agent? Let’s find out.

Proving the metric catches failure: a broken-retriever regression#

The most convincing test of a success metric is to break the agent on purpose and confirm the number moves. We change nothing about the agent’s logic, the tasks, or the verifier. We only swap in a search_docs_broken that returns the worst-matching pages instead of the best – a stand-in for a real retrieval regression, like a bad embedding-model swap that quietly starts returning garbage. Because the agent now reasons over irrelevant context, its answers should stop containing the reference facts, and the verifier should catch every one.

def search_docs_broken(query, docs, k=2):
    """A DELIBERATELY broken retriever: returns the WORST-matching pages (lowest keyword overlap),
       simulating a retrieval regression. The agent code is unchanged."""
    q = set(query.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())))
    return [{"id": d["id"], "title": d["title"], "text": d["text"]} for d in scored[:k]]

rows_bad = run_suite(TASKS, retriever=search_docs_broken)
report(rows_bad, "SAME agent, broken retriever injected")
for r in rows_bad:
    if not r["ok"] and r["kind"] == "fact":
        print(f"  caught failure -> {r['q'][:52]}\n     agent said: {r['answer'][:120]}")

== SAME agent, broken retriever injected ==
task                                                          type   result
----------------------------------------------------------------------------------
What HTTP status code does Meridian return when you exce      fact   FAIL
How much does the Pro plan cost per month?                    fact   FAIL
Can you change a database's region after it is created?       fact   FAIL
How long are backups retained on the Scale plan?              fact   FAIL
Which environment variable does the Python SDK read for       fact   FAIL
What does a 401 error mean?                                   fact   FAIL
How long is the grace period before a deleted database i      fact   FAIL
Does Meridian support MongoDB-style aggregation pipeline    refuse   PASS
----------------------------------------------------------------------------------
Task success rate = 1/8 = 0.125

  caught failure -> What HTTP status code does Meridian return when you 
     agent said: I don't have that in the docs. The search results didn't return any information about rate limiting or the HTTP status c
  caught failure -> How much does the Pro plan cost per month?
     agent said: I don't have that in the docs. The search results didn't return any pricing information for a Pro plan. You may need to 
  caught failure -> Can you change a database's region after it is creat
     agent said: I don't have that information in the docs. The search results aren't returning documentation about changing a database's
  caught failure -> How long are backups retained on the Scale plan?
     agent said: I don't have that information in the docs. The search results don't contain details about backup retention periods for t
  caught failure -> Which environment variable does the Python SDK read 
     agent said: (stopped: max steps)
  caught failure -> How long is the grace period before a deleted databa
     agent said: I don't have that in the docs. The search results don't contain information about the grace period for deleted databases

Task success rate collapsed from 8/8 to 1/8 – the outcome metric caught the regression instantly, with zero changes to the eval code. Read the captured failures and you can see how the agent failed: starved of the right pages, it mostly did the honest thing and refused (“I don’t have that in the docs”) – which is correct behavior for a question it genuinely can’t answer from the (broken) context, but a failure of the task, because the fact was retrievable and the user still didn’t get their answer. One task even hit max_steps and gave up entirely. Only two tasks survived: the out-of-scope MongoDB probe (whose success criterion is a refusal, so a refusal passes), and, on this run, nothing else.

This is the whole argument for outcome metrics in one before-and-after. A wrong retriever would sail past a superficial “the agent responded and it sounds fluent” glance – every one of those refusals reads as calm and reasonable. The task success rate is not fooled: it compares the outcome against a reference and reports the collapse as a hard number you can alert on.

Binary vs partial credit, made concrete#

Finally, make the binary-versus-partial distinction concrete on a two-part task. We score the agent’s real answer to “What does the Free plan cost per month, and how much storage does it include?” against two sub-goals – the price and the storage – then score a hypothetical half-right answer to show where the two scoring schemes diverge. The scoring function is a pure keyword check, so given a fixed answer string it is fully deterministic and reproducible.

SUBGOALS = {"price": ["0 dollars", "free", "no cost", "zero", "$0"], "storage": ["1 gb", "1gb"]}

def score_answer(answer):
    a = answer.lower()
    hits = {name: any(k in a for k in kws) for name, kws in SUBGOALS.items()}
    binary = 1.0 if all(hits.values()) else 0.0
    partial = sum(hits.values()) / len(hits)
    return hits, binary, partial

# 1) the real agent answering a 2-part question (live)
ans, _ = agentic_docent("What does the Free plan cost per month, and how much storage does it include?", DOCS)
hits, binary, partial = score_answer(ans)
print("real agent answer:", " ".join(ans.split())[:110])
print(f"  sub-goals met = {hits}  ->  binary = {binary:.2f}  partial = {partial:.2f}")

# 2) a HYPOTHETICAL half-right answer, to show where the two scores diverge (deterministic)
half = "The Free plan costs 0 dollars per month."   # mentions price, omits storage
hits2, binary2, partial2 = score_answer(half)
print("half-right answer:", half)
print(f"  sub-goals met = {hits2}  ->  binary = {binary2:.2f}  partial = {partial2:.2f}")

real agent answer: Based on the documentation, the **Free plan costs $0 per month** and includes **1 GB of storage**.
  sub-goals met = {'price': True, 'storage': True}  ->  binary = 1.00  partial = 1.00
half-right answer: The Free plan costs 0 dollars per month.
  sub-goals met = {'price': True, 'storage': False}  ->  binary = 0.00  partial = 0.50

The real agent nailed both sub-goals, so binary and partial agree at 1.00. The half-right answer is where they part ways: it gets the price but drops the storage, so binary scores it a flat 0.00 while partial credit scores 0.50. Neither is “right” in the abstract – they answer different questions. Binary is honest about the user experience (an answer missing half of what was asked is not a complete success), while partial credit gives you a finer gradient for tracking whether a change nudged the agent closer to fully solving multi-part tasks. Choose per what you’re deciding: ship/no-ship leans binary; iterating on a hard multi-step workflow leans partial.

Practice Exercises#

Exercise 1: Add a wrongly-refused factual task#

Add a factual task whose accept fact really is in the docs – say {"q": "How many requests per minute does the Free plan allow?", "accept": ["60"], "out_of_scope": False} – then run it through the broken retriever. Confirm the verifier marks it FAIL even though the agent’s refusal (“I don’t have that in the docs”) sounds perfectly reasonable. Explain why counting a plausible-sounding refusal as a failure is the correct behavior for an outcome metric.

Exercise 2: Compute a confidence-style range over repeated runs#

Because the run is non-deterministic, a single 8/8 can flatter the agent. Run the good-retriever suite three times, record the success rate each time, and report the min, max, and mean. Discuss why reporting a range (or the number of runs) is more honest than quoting one run’s rate, and how many golden tasks you’d want before a single flipped task stops swinging the rate so much.

Exercise 3: Turn the verifier into an LLM judge for an open-ended task#

Keyword accept lists work for factual answers, but they break on open-ended ones. Add a task like “Summarize how Meridian handles rate limiting” where no single keyword captures “correct,” and replace the keyword check for that task with a tiny LLM-judge call: prompt claude-haiku-4-5 with the answer and a rubric (“Does the summary mention the 429 status code and that limits are per API key? Reply PASS or FAIL”) and parse its verdict. Explain the trade-off you just made.

Summary#

Task success rate is the headline outcome metric for an agent: the fraction of tasks it completed correctly, where “correctly” is made checkable by a per-task verifier. You defined a task_succeeded verifier for the agentic Docent – a keyword check that a factual answer contains the reference fact, a refusal check that the out-of-scope task is correctly declined, and a guard that fails any factual task the agent wrongly refuses – and ran the agent live over 8 golden tasks for a real 8/8 = 1.000. Because a metric you never see fail is untrustworthy, you injected a deliberately broken retriever into the same agent and watched success collapse to 1/8: starved of the right pages, the agent mostly refused, and the outcome metric caught every one of those plausible-sounding failures without a single change to the eval code. You separated outcome from process (a successful agent can still be wasteful – that’s Lesson 4), chose among exact end-state checks (for state-changing agents), verifier functions (Docent’s case, since it’s read-only), and LLM judges (for open-ended tasks), and weighed binary success against partial credit, seeing a half-right two-part answer score a flat 0 under binary but 0.5 under partial. The live numbers vary run to run, which is exactly why you evaluate over a set and watch for movement from a baseline rather than trusting a single question once.

Key Concepts#

• Task success rate – $\text{completed correctly} / \text{total tasks}$; the headline outcome metric. Docent scored 8/8 live on the good retriever, 1/8 on a broken one.
• Success criterion / verifier – the per-task rule that returns pass or fail. For Docent: reference-fact keyword match for factual tasks, correct refusal for the out-of-scope task.
• End-state verification – for agents that change the world, check the world afterward, not the transcript. Docent is read-only, so it uses a final-answer verifier instead.
• Verifier function vs LLM judge – deterministic keyword/regex/schema checks when the answer has a checkable signature; a judge model for open-ended answers that no keyword captures.
• Binary vs partial credit – all-or-nothing per task vs the fraction of sub-goals met; a half-right two-part answer scores 0.00 binary, 0.50 partial.
• Outcome vs process – task success measures whether the agent achieved the goal, independent of how efficiently; cost and steps are a separate axis (Lesson 4).

Why This Matters#

The task success rate is the number a stakeholder actually asks for – “does the agent work?” – and it’s the one an ordinary demo hides. A broken retriever produced eight calm, fluent, entirely wrong responses; “looks fine” would have shipped it. The outcome metric refused to be fooled, dropping from 8/8 to 1/8 the moment the agent stopped delivering the reference facts, which is exactly the alarm you want wired to a regression. But the metric is only as good as its verifier: too loose and it rubber-stamps failures, too strict and it fails correct answers, and for open-ended work you’ll need a calibrated judge instead of a keyword list. Getting the success criterion right – and reporting the rate honestly as a range over repeated runs rather than a single flattering number – is what turns “did it work?” from a gut feeling into a metric you can track, alert on, and trust.

Continue Building Your Skills#

You now have the headline agent metric working for real: a task success rate computed from a per-task verifier, proven to catch a regression by dropping from 8/8 to 1/8 the instant the agent’s retriever broke. That number tells you whether Docent accomplished the task – but not how. An agent can reach the right answer down a wrong path (a mis-called tool that happened not to matter) or a wasteful one (four searches where one would do), and a success-only view is blind to both. The next lesson opens up the mechanism: tool-call correctness asks whether Docent called search_docs at all, with a sensible query, and handled the result properly – catching failures that hide inside a task that still, sometimes, comes out right.