Lesson 2 - Offline vs. Online Evaluation

Welcome to Offline vs. Online Evaluation#

In Lesson 1, reading a handful of Docent’s answers made it look finished, and then a question you never happened to check revealed a failure. The fix was to stop reading answers one at a time and start measuring them against something. This lesson names the two places that measurement can happen, because they are genuinely different jobs with different tools.

Offline evaluation runs Docent over a fixed set of questions you control, before the change reaches a single user. Online evaluation watches Docent answer real questions from real users, after it ships. Docent is our documentation Q&A assistant for Meridian, the serverless database, and both regimes apply to it directly: offline, you score it on a curated golden set drawn from the Meridian docs; online, you log the questions developers actually ask and watch which answers get a thumbs-down. Neither one replaces the other, and this lesson is about why you need both.

In this lesson, you will:

• Define offline evaluation and run a real offline pass over four golden questions for Docent
• Define online evaluation and identify the implicit signals real traffic provides
• Explain how offline gates a release and online feeds new cases back into the offline set, forming a loop
• Reason about the core trade-offs: coverage versus realism, speed versus signal, reproducibility versus relevance

Offline Evaluation: Measure Before You Ship#

Offline evaluation means running your app over a fixed dataset and scoring the results, with no live users involved. The dataset is a golden set: a curated list of inputs paired with what a correct output should contain. Because the inputs never change between runs, an offline evaluation is reproducible — the same prompts and the same model give you the same score, so a number moving up or down is caused by your change, not by luck.

That property is what makes offline evaluation the tool for comparison and gating. Want to know whether a reworded prompt helps? Run the golden set on the old prompt, run it on the new prompt, compare the two scores. Want to know whether claude-haiku-4-5 is good enough before you ship? Score it offline and set a bar the release must clear. Offline runs are fast and cheap enough to do on every change, which is exactly what you want from a release gate.

For Docent, the golden set is short questions about Meridian whose answers we already know from the docs. Here is a real offline pass over four of them. Each item carries a must_include token — the fact a correct answer has to mention — and a tiny grader checks for it. We reuse the canonical Docent implementation from the module.

import warnings
warnings.filterwarnings("ignore")
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

DOCS = [
    {"id": "ratelimits", "title": "Rate Limits",
     "text": "The Free plan allows 60 requests per minute. The Pro plan allows 600 requests per minute. Exceeding the "
             "limit returns HTTP 429 with a Retry-After header giving the seconds to wait. Rate limits are applied per "
             "API key, not per account."},
    {"id": "plans", "title": "Plans and Pricing",
     "text": "Meridian has three plans. Free costs 0 dollars per month and includes 1 GB of storage. Pro costs 25 dollars "
             "per month and includes 50 GB of storage. Scale costs 199 dollars per month and includes 500 GB of storage. "
             "All plans include unlimited read replicas."},
    {"id": "regions", "title": "Regions",
     "text": "Databases can be created in three regions: us-east, eu-west, and ap-south. The region is fixed at creation "
             "time and cannot be changed later; to move data to another region you must export and re-import into a new "
             "database. Cross-region read replicas are available on the Scale plan only."},
    {"id": "sdks", "title": "SDKs",
     "text": "Official SDKs are available for Python, JavaScript, and Go. Install the Python SDK with 'pip install meridian'. "
             "The SDK reads the MERIDIAN_API_KEY environment variable by default. All SDKs retry idempotent requests up to "
             "3 times on 5xx errors with exponential backoff."},
]

def retrieve(question, docs, k=2):
    q = set(question.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())), reverse=True)
    return scored[:k]

def docent_answer(question, docs, model="claude-haiku-4-5"):
    ctx = "\n\n".join(f"[{d['id']}] {d['title']}\n{d['text']}" for d in retrieve(question, docs))
    prompt = (f"Answer the question using ONLY the context. If the context does not contain the answer, say "
              f"\"I don't have that in the docs.\" Be concise.\n\nContext:\n{ctx}\n\nQuestion: {question}")
    msg = client.messages.create(model=model, max_tokens=200,
                                 messages=[{"role": "user", "content": prompt}])
    return msg.content[0].text

# The golden set: fixed inputs, each with the fact a correct answer must mention.
GOLDEN = [
    {"q": "What HTTP status code does Meridian return when you exceed the rate limit?", "must_include": "429"},
    {"q": "How many requests per minute does the Pro plan allow?",                      "must_include": "600"},
    {"q": "How much does the Pro plan cost per month?",                                 "must_include": "25"},
    {"q": "Which environment variable does the Python SDK read for the API key?",       "must_include": "MERIDIAN_API_KEY"},
]

def grade(answer, must_include):
    return must_include.lower() in answer.lower()

passed = 0
for item in GOLDEN:
    answer = docent_answer(item["q"], DOCS)
    ok = grade(answer, item["must_include"])
    passed += ok
    print(f"[{'PASS' if ok else 'FAIL'}] must contain {item['must_include']!r:>18}  <-  {answer.strip()[:45]!r}")

print(f"\nAggregate: {passed}/{len(GOLDEN)} passed  (score = {passed / len(GOLDEN):.2f})")

[PASS] must contain              '429'  <-  'HTTP 429'
[PASS] must contain              '600'  <-  'The Pro plan allows 600 requests per minute.'
[PASS] must contain               '25'  <-  'The Pro plan costs $25 per month.'
[PASS] must contain 'MERIDIAN_API_KEY'  <-  'The Python SDK reads the `MERIDIAN_API_KEY` e'
Aggregate: 4/4 passed  (score = 1.00)

Docent scores 4 out of 4 on this golden set. The wording of each answer is a real Claude response and will vary slightly if you re-run it — “HTTP 429” might come back as “Meridian returns HTTP 429.” — but the grade does not vary, because the grader only checks whether the required fact is present. That separation is what makes offline evaluation a dependable gate: the model output is fuzzy, but the score is reproducible.

Online Evaluation: Measure After You Ship#

Online evaluation means measuring Docent on real production traffic, after it is live. You no longer control the inputs — real users do — so you cannot pre-write a must_include for each one. Instead you rely on implicit signals that real usage generates for free, plus a bit of instrumentation you add on purpose:

• Explicit feedback: a thumbs-up or thumbs-down button under each answer.
• Behavioral signals: a user who edits Docent’s answer before using it, asks the same question again reworded, or abandons the session — all hint the answer missed.
• Guardrail and refusal rates: how often Docent says “I don’t have that in the docs.” A sudden spike might mean retrieval broke; a drop to zero might mean it stopped refusing questions it should refuse.
• Sampled judging: log a random slice of live answers and score them later, offline-style, with a human or an LLM judge.
• A/B tests: ship two prompt variants to different users and compare their live thumbs-up rates.

Here is an illustrative live stream — two real questions run through Docent, paired with hand-written mock thumbs standing in for what a user might click. The thumbs values are not model output; they are there to show the shape of an online signal.

# The thumbs values below are HAND-WRITTEN illustrative user feedback, not model output.
LIVE_STREAM = [
    {"user_q": "how do I move my database to another region?",           "thumb": "down"},
    {"user_q": "does meridian support mongodb aggregation pipelines?",   "thumb": "up"},
]

ups = 0
for event in LIVE_STREAM:
    answer = docent_answer(event["user_q"], DOCS)
    ups += (event["thumb"] == "up")
    print(f"user: {event['user_q']}")
    print(f"Docent: {answer.strip()[:60]!r}")
    print(f"feedback (mock): thumbs-{event['thumb']}\n")

print(f"Live thumbs-up rate (illustrative): {ups}/{len(LIVE_STREAM)} = {ups / len(LIVE_STREAM):.2f}")

user: how do I move my database to another region?
Docent: 'To move your database to another region, you must export a'
feedback (mock): thumbs-down

user: does meridian support mongodb aggregation pipelines?
Docent: "I don't have that in the docs."
feedback (mock): thumbs-up

Live thumbs-up rate (illustrative): 1/2 = 0.50

Look at what online caught that offline could not. The first question is factually correct — Meridian really does require an export and re-import to change regions — yet the user pressed thumbs-down, perhaps because they wanted step-by-step instructions and got a one-line summary. No must_include check would have flagged that; only a real person reacting to a real answer surfaces it. The second is the out-of-scope MongoDB probe: Docent correctly refuses, and the (mock) user is satisfied. Online evaluation is noisier and needs live traffic to produce any signal at all, but it measures the thing offline can only approximate — whether real people are actually helped.

How They Fit Together: One Loop#

Offline and online are not competitors; they are two halves of a cycle. Offline runs first and acts as the release gate: if Docent’s score on the golden set drops below your bar, the change does not ship. Once it clears the bar and goes live, online takes over and watches real traffic for the failures the golden set never contained. The crucial part is what happens next: every failure online discovers — that thumbs-down on the region question — gets turned into a new golden-set item, with a reference answer written for it. The offline set grows to cover exactly the gaps production revealed, so the next release is gated on a harder, more realistic bar.

The three trade-offs in that caption are worth stating plainly, because they explain why you cannot pick just one regime:

• Coverage vs. realism. Offline covers exactly the cases you curated — broad if you work at it, but always artificial. Online only sees whatever users happen to send, but every one of those is real.
• Speed vs. signal. Offline gives you a score in seconds, on demand, before shipping. Online gives you a better signal, but slowly, and only after real people have used the product.
• Reproducibility vs. relevance. Offline is reproducible, so it can attribute a score change to your edit. Online is relevant, so it measures what actually matters — at the cost of noise you have to average away over many sessions.

Practice Exercises#

Exercise 1: Add a golden item that could fail#

The offline pass above scored 4 out of 4 because every question maps cleanly to a fact in the Meridian docs. Add a fifth golden item for the question “How long are backups retained on the Scale plan?” with the appropriate must_include token, and describe what you would need to add to DOCS for Docent to have any chance of answering it.

Exercise 2: Classify each online signal#

For each of these production observations about Docent, decide whether it is an explicit signal, a behavioral signal, or a guardrail/refusal-rate signal: (a) a user clicks thumbs-down; (b) Docent’s “I don’t have that in the docs.” rate jumps from 8% to 40% overnight; (c) a user rephrases the same question three times in one session.

Exercise 3: Match the regime to the decision#

For each decision, say whether offline or online evaluation is the right tool and why: (1) choosing between two prompt wordings before launch; (2) deciding whether a change that shipped last week actually helped real users; (3) proving that a score dropped because you switched models and not because of random variation.

Summary#

Offline evaluation runs Docent over a fixed golden set before shipping. It is reproducible, fast, and cheap, which makes it the right tool for comparing prompts and models and for gating a release — in this lesson Docent scored a real 4 out of 4 on four golden questions, with the grade staying stable even though the exact answer wording varied. Its limit is that it can only ever test the cases you thought to include. Online evaluation measures Docent on real production traffic after shipping, using implicit signals like thumbs, edits, follow-up questions, refusal rates, and A/B tests; the illustrative live stream showed a factually-correct answer still earning a thumbs-down, a failure no offline check would have caught. The two form a loop: offline gates the release, online catches what offline missed, and every online failure becomes a new golden-set item that hardens the next gate. Choosing between them comes down to three trade-offs — coverage versus realism, speed versus signal, and reproducibility versus relevance — and the answer is almost always to run both.

Key Concepts#

• Offline evaluation — running the app over a fixed golden set before shipping; reproducible and cheap, used to compare and to gate releases.
• Golden set — a curated list of inputs paired with what a correct output must contain; the fixed dataset an offline eval scores against.
• Online evaluation — measuring the app on real production traffic after shipping, via implicit signals and sampling.
• Implicit signal — feedback that real usage generates for free, such as thumbs, edits, rephrasings, abandonment, and refusal rates.
• The evaluation loop — offline gates the release; online surfaces new failures; those failures become new golden-set items that strengthen the next offline gate.

Why This Matters#

If Docent shipped on offline scores alone, a 4-out-of-4 golden set would give a false sense of safety while real developers hit region and backup questions the set never included. If it shipped on online signals alone, every regression would be discovered by a frustrated user, with no gate to catch bad changes first. Running both is what lets a team change Docent’s prompt or swap its model confidently: offline stops the obvious regressions before anyone sees them, and online quietly reports which real questions still go unanswered — feeding them back so the golden set, and the product, keep getting better.

Continue Building Your Skills#

You can now tell offline evaluation apart from online evaluation, run a reproducible offline pass that scores Docent on a golden set, read the implicit signals real traffic provides, and explain how the two regimes form a loop that keeps a golden set honest. The next lesson builds on this by developing the evaluation mindset itself — the shift from asking “does this answer look good?” to “what would I measure, on what set, to know?” — so that every change to Docent is judged by a number you can trust rather than a feeling you cannot.