Lesson 3 - The Evaluation Mindset

Welcome to the Evaluation Mindset#

In Lessons 1 and 2 you watched Docent, our documentation Q&A assistant for the Meridian serverless database, sail through a handful of hand-picked questions and then quietly stumble on one you didn’t happen to check. The problem was never Docent’s answers. The problem was the method: reading a few outputs and forming an impression. That method is called “vibes,” and it does not scale, does not reproduce, and does not tell you whether yesterday’s change made things better or worse.

This lesson replaces vibes with a repeatable measurement. You will learn the anatomy of a real evaluation — the four parts every eval is built from — and the loop that runs them. Then you’ll build a small but genuine eval harness in Python and run it on Docent’s task. Crucially, the harness runs with no API key and gives the same numbers every time, because reproducibility is the whole point of the mindset shift.

In this lesson, you will:

• Name the four components every evaluation needs: a task, a dataset, a metric, and a target with an aggregation rule.
• Walk the eval loop — run, score, aggregate, compare, decide — and see why fixing the dataset and versioning the prompt make it reproducible.
• Tell an absolute eval (score against a threshold) apart from a regression eval (better or worse than last time).
• Build a real, generic run_eval(system_fn, dataset, metric_fn) and demonstrate it with a deterministic metric that needs no model call.
• Spot the beginner mistakes that make an eval lie to you.

From “looks fine” to a number you can defend#

“Vibes” evaluation has a specific failure mode: it is not written down. When you read five of Docent’s answers and conclude “good enough,” nobody — including future you — can rerun that judgement, disagree with it precisely, or check whether next week’s prompt tweak preserved it. An evaluation is the opposite. It is a written-down procedure that takes your system and returns a number (or a small set of numbers) that you can store, compare, and argue about.

The mindset shift is this: stop asking “does this answer look right?” and start asking “what score does my system get on a fixed set of questions, measured by a rule I decided in advance?” The second question has an answer that doesn’t depend on your mood, how many examples you had the patience to read, or which examples you happened to pick.

Every evaluation that earns that trust is assembled from the same four parts.

The four components of an evaluation#

Whether you are grading Docent or a fraud classifier, an eval is built from four pieces. Get all four explicit and you have a measurement; leave one implicit and you have vibes wearing a lab coat.

1. A task definition — what “good” means. Before you can score anything you must decide what the system is for and what a good output looks like. For Docent the task is: given a developer’s question about Meridian, return a short, correct answer grounded in the docs — and when the docs don’t cover the question, say so instead of guessing. That single sentence already rules out some answers (a confident wrong number) and rules in others (an honest “I don’t have that in the docs”). If you can’t write the task down, you can’t measure it.

2. A dataset of examples. You need concrete inputs to run the system on, and — wherever you can get them — reference outputs that say what a good answer contains. For Docent an example is a question plus the short known-correct answer, like “What HTTP status code does Meridian return when you exceed the rate limit?” with reference 429. The dataset must be fixed: the same items, in the same form, every time you run. A dataset that changes underneath you turns every score into an apples-to-oranges comparison.

3. A metric — a function that scores an output. A metric is literally a function: it takes the system’s output (and usually the reference) and returns a score. Metrics range from trivial (does the reference string appear in the answer?) to sophisticated (a second model judging faithfulness — later in the course). What matters right now is that a metric is code you can run, not an impression you form.

4. A target and an aggregation rule. One score per item isn’t a decision. You need a way to aggregate the per-item scores into a summary — often a simple mean, sometimes a pass rate or a worst-case — and a target/threshold that says what counts as good enough (for example, “at least 90% of items must pass”). The aggregation rule and the target are choices; make them on purpose, because they decide what your eval rewards.

The eval loop#

With the four components in hand, running an evaluation is a fixed loop:

1. Run the system over every item in the dataset, collecting one output per input.
2. Score each output with the metric, producing one number per item.
3. Aggregate the per-item scores into a summary (a mean, a pass rate).
4. Compare the aggregate against a threshold, or against a previous version’s aggregate.
5. Decide: ship it, or send it back for a fix.

The loop is worthless unless it is reproducible. Three habits guarantee that:

• Fix the dataset. Same examples, every run. If you add items, that’s a new version of the dataset — note it.
• Version the prompt (and the model). Record which prompt and which model produced a score. A number without its prompt is unattributable.
• Record the score. Write it down next to the version that produced it, so tomorrow’s run has something to compare against.

Do those three and the loop becomes a measuring instrument. Skip them and you’re back to vibes with extra steps.

Building a real eval harness#

Time to make this concrete. We’ll build a generic harness — run_eval(system_fn, dataset, metric_fn) — that knows nothing about Docent, Claude, or any particular metric. That generality is deliberate: later modules will plug real metrics and the live Docent assistant into exactly this shape. Right now we demonstrate it with a deterministic metric and a stand-in system so the whole thing runs offline and reproduces exactly.

The harness itself is small. It runs the system over each item, scores each output, and aggregates:

def run_eval(system_fn, dataset, metric_fn):
    per_item = []
    for i, item in enumerate(dataset):
        output = system_fn(item["input"])
        score = metric_fn(output, item.get("reference"))
        per_item.append({
            "id": item.get("id", i),
            "output": output,
            "reference": item.get("reference"),
            "score": score,
        })
    scores = [r["score"] for r in per_item]
    aggregate = round(sum(scores) / len(scores), 4) if scores else 0.0
    return per_item, aggregate

# A trivial, deterministic metric: does the reference answer appear in the output?
def contains_reference(output, reference):
    if reference is None:
        return 0.0
    return 1.0 if reference.lower() in output.lower() else 0.0

# A tiny in-memory dataset (fixed, versioned, reproducible).
EVAL_SET = [
    {"id": "q1", "input": "What HTTP status code does Meridian return when you exceed the rate limit?",
     "reference": "429"},
    {"id": "q2", "input": "How many requests per minute does the Pro plan allow?",
     "reference": "600 requests per minute"},
    {"id": "q3", "input": "Which environment variable does the Python SDK read for the API key?",
     "reference": "MERIDIAN_API_KEY"},
    {"id": "q4", "input": "How long is the grace period before a deleted database is permanently purged?",
     "reference": "24 hours"},
]

# A stand-in "system": a fixed lookup table, so the demo needs no API key.
# In production this is Docent (which calls Claude); here it is deterministic
# so the whole eval runs offline and returns identical numbers every time.
CANNED = {
    "What HTTP status code does Meridian return when you exceed the rate limit?":
        "Meridian returns HTTP 429 when the rate limit is exceeded.",
    "How many requests per minute does the Pro plan allow?":
        "The Pro plan allows 600 requests per minute.",
    "Which environment variable does the Python SDK read for the API key?":
        "It reads the MERIDIAN_API_KEY environment variable.",
    "How long is the grace period before a deleted database is permanently purged?":
        "There is a 24-hour grace period, after which the data is purged.",
}

def canned_system(question):
    return CANNED.get(question, "I don't have that in the docs.")

per_item, aggregate = run_eval(canned_system, EVAL_SET, contains_reference)
for r in per_item:
    print(f"{r['id']}: score={r['score']}  ref={r['reference']!r}")
passed = int(sum(r["score"] for r in per_item))
print(f"aggregate = {aggregate}  ({passed}/{len(per_item)} passed)")

Running it:

q1: score=1.0  ref='429'
q2: score=1.0  ref='600 requests per minute'
q3: score=1.0  ref='MERIDIAN_API_KEY'
q4: score=0.0  ref='24 hours'
aggregate = 0.75  (3/4 passed)

Run it a second time and you get the exact same output — no randomness, no API call, no drift. That is what reproducibility feels like, and it is the whole reason we started with a deterministic metric.

Now look at q4. Its score is 0.0, yet the system’s answer — “There is a 24-hour grace period, after which the data is purged” — is completely correct. The reference is 24 hours; the answer says 24-hour. The substring metric can’t see past the hyphen. That’s not a bug in the harness; it is a metric that doesn’t perfectly reflect the task, caught in the act. We’ll keep it precisely because it is honest: cheap string-match metrics are fast and reproducible, but they punish correct answers that are phrased differently, so you must read their failures rather than trust the number blindly. Later modules replace it with metrics that understand meaning.

Absolute versus regression evals#

The same harness answers two different questions, and it’s worth knowing which one you’re asking.

An absolute eval scores your system against a fixed threshold: is Docent good enough to ship? You pick a target — say, at least 90% of items must pass — and compare the aggregate to it. The answer is a yes/no about this version, in isolation.

A regression (or relative) eval compares your system against a baseline — usually the previous version — on the same dataset: did this change make things better or worse? You don’t need an absolute bar for this; you need last version’s score. This is the question you ask on every prompt tweak, every model swap, every retrieval change.

Here we ship a “version 2” of the system that phrases the deletion answer as “24 hours,” and ask both questions at once. This block continues from the previous one, reusing run_eval, contains_reference, EVAL_SET, and canned_system:

# Version 2: identical to v1 except the deletion answer now says "24 hours"
# (v1 said "24-hour", which the substring metric could not match).
CANNED_V2 = {
    "What HTTP status code does Meridian return when you exceed the rate limit?":
        "Meridian returns HTTP 429 when the rate limit is exceeded.",
    "How many requests per minute does the Pro plan allow?":
        "The Pro plan allows 600 requests per minute.",
    "Which environment variable does the Python SDK read for the API key?":
        "It reads the MERIDIAN_API_KEY environment variable.",
    "How long is the grace period before a deleted database is permanently purged?":
        "A deleted database has a grace period of 24 hours before it is purged.",
}

def system_v2(question):
    return CANNED_V2.get(question, "I don't have that in the docs.")

TARGET = 0.9  # absolute bar: at least 90% of items must pass

# Absolute eval: does each version clear the bar on its own?
_, agg_v1 = run_eval(canned_system, EVAL_SET, contains_reference)
_, agg_v2 = run_eval(system_v2, EVAL_SET, contains_reference)

print(f"v1 aggregate = {agg_v1}   passes target {TARGET}? {agg_v1 >= TARGET}")
print(f"v2 aggregate = {agg_v2}   passes target {TARGET}? {agg_v2 >= TARGET}")

# Regression eval: is v2 better, worse, or the same as v1 on the SAME set?
delta = round(agg_v2 - agg_v1, 4)
verdict = "improvement" if delta > 0 else "regression" if delta < 0 else "no change"
print(f"delta (v2 - v1) = {delta:+}  ->  {verdict}")

Output:

v1 aggregate = 0.75   passes target 0.9? False
v2 aggregate = 1.0   passes target 0.9? True
delta (v2 - v1) = +0.25  ->  improvement

Read both answers. The absolute eval says v1 falls short of the 90% bar while v2 clears it. The regression eval says v2 is a +0.25 improvement over v1 on the same fixed set. Notice that the only thing that changed was the phrasing of one answer — the dataset and the metric were held constant — so the delta is attributable to that one change. That is the payoff of a fixed dataset and a versioned system: the number moved for a reason you can name.

The beginner mistakes that make an eval lie#

An evaluation is only as trustworthy as its weakest component. Four mistakes show up constantly:

• No fixed dataset. Grading a different handful of questions each run means every score is incomparable. If the set can change, it isn’t an eval — it’s sampling your own optimism.
• Changing the prompt and the test at the same time. If you edit Docent’s prompt and add three new questions before rerunning, and the score moves, you have no idea which change did it. Move one variable at a time.
• A metric that doesn’t reflect the task. Our substring metric scored a correct “24-hour” answer as a failure. A metric that rewards the wrong thing (length, keyword stuffing, exact phrasing) will happily certify a worse system. Sanity-check what your metric actually measures.
• Averaging away important failures. A mean of 0.9 can hide a 0.0 on the one question that matters most — say, Docent confidently inventing a rate limit. When some failures are worse than others, aggregate accordingly (a worst-case, a per-category pass rate) instead of letting the average bury them.

None of these are exotic. All of them are avoidable the moment you treat the four components as things you decide on purpose rather than defaults you drift into.

Practice Exercises#

Exercise 1: Add a refusal example and see the metric strain#

Docent’s task includes refusing questions the docs can’t answer. Add a fifth item to EVAL_SET for the out-of-scope question “Does Meridian support MongoDB-style aggregation pipelines?” with a reference phrase you’d accept as a correct refusal (for example, "don't have that in the docs"). Add a matching entry to CANNED that refuses, then rerun run_eval and report the new aggregate. Does the substring metric handle the refusal cleanly, and what does that tell you about relying on exact phrases?

Exercise 2: Swap the aggregation rule to expose a hidden failure#

The default aggregate is a mean, which can hide a single critical failure. Write a second aggregation over the same per_item list that reports the worst-case score (the minimum) alongside the mean. Construct a small dataset where the mean looks healthy (say 0.8) but the minimum is 0.0 on an item you care about, and print both. Which number would you gate a release on, and why?

Exercise 3: Run a regression eval that catches a real regression#

The lesson’s regression eval showed an improvement. Now force a regression: build a system_v3 that answers three items correctly but gets one previously-correct item wrong (return the wrong number, or a blanket “I don’t have that in the docs.”). Run the same absolute-plus-regression comparison of v2 versus v3 and confirm the delta is negative and the verdict reads regression. Then state, in one sentence, what a negative delta should trigger in a real workflow.

Summary#

You traded impressions for measurement. An evaluation is not a feeling about a few answers; it is four explicit components — a task, a dataset, a metric, and a target with an aggregation rule — run through a fixed loop that produces a number you can store and compare. You built that loop as a generic run_eval harness, demonstrated it on Docent’s task with a deterministic metric that reproduces exactly, and used it to ask both an absolute question (is this good enough?) and a regression question (is this better than last time?).

Key Concepts#

The four components

• Every eval needs a task (what “good” means), a dataset (fixed inputs, ideally with reference outputs), a metric (a function that scores an output), and a target plus aggregation rule (a threshold and how you roll scores up).
• Leave any one implicit and you’re back to vibes.

The eval loop and reproducibility

• Run over the dataset, score each output, aggregate, compare against a threshold or baseline, decide.
• It only measures if you fix the dataset, version the prompt and model, and record the score — our deterministic demo returned aggregate = 0.75 on every run.

Absolute versus regression evals

• An absolute eval compares the aggregate to a threshold (v2 cleared the 0.9 bar; v1 didn’t).
• A regression eval compares against a baseline on the same set (v2 was +0.25 over v1) and is what you run on every change.

The beginner mistakes

• No fixed dataset; changing prompt and test together; a metric that doesn’t reflect the task (our q4 scored a correct answer 0.0); averaging away a critical failure.

Why This Matters#

The mindset in this lesson is the foundation the rest of the course stands on. Every later technique — model-graded metrics, faithfulness checks, retrieval evaluation, production monitoring — is a better metric or a bigger dataset dropped into exactly the loop you just built. Because you separated the harness from the metric and the system, you can upgrade any one part without rewriting the others: swap the deterministic metric for an LLM judge, swap the canned system for live Docent on claude-haiku-4-5, grow the four-item set into a real suite, and run_eval doesn’t change. That separation is what lets a team improve an LLM app deliberately instead of hoping each change was an improvement. And the reproducibility habit — same dataset, versioned prompt, recorded score — is what turns “I think it got better” into “it went from 0.75 to 1.0 on the eval set,” which is the only kind of claim you can actually defend.

Continue Building Your Skills#

The habit worth carrying out of this lesson is smaller than any metric: whenever you judge an LLM app, ask yourself which of the four components you’ve actually pinned down. Do you have a written task definition, or just a hunch about what’s good? A fixed dataset, or whichever examples you tried today? A metric that’s real code, or an impression? A target and an aggregation rule you chose on purpose, or a mean you fell into? Every time you can answer all four concretely, you’ve turned a vibe into a measurement — and everything the rest of this course teaches is a way to make one of those four components sharper without touching the loop that binds them together.