Lesson 2 - Writing Test Cases by Hand

Welcome to Writing Test Cases by Hand#

In Lesson 1 you learned what separates a trustworthy golden set from a handful of cherry-picked questions: representativeness, labeled references, diversity, and hard cases held out on purpose. That was the specification. This lesson is where you actually build it, one case at a time, by hand.

Hand-writing test cases sounds like the least glamorous part of evaluation, and it is the part that most decides whether your numbers mean anything. A metric can only be as good as the case it scores. If a case has no reference, nothing can grade it. If every case is an easy happy-path question, a broken Docent still scores well. If the reference captures one phrasing instead of the underlying fact, a correct answer fails for saying the right thing the wrong way. The skill here is authoring cases that are structured, checkable, and deliberately varied — so that when Docent, our documentation assistant for the Meridian serverless database, passes your set, that actually means something.

By the end you will have a real, validated dataset of ten Docent cases spanning five case types, plus a schema that refuses to let a malformed case in. Everything in this lesson is deterministic — no model call, no API key — so it runs identically every time.

In this lesson, you will:

• Name the seven fields that make up a well-formed test case and what each one is for
• Deliberately include five case types: factual, paraphrase, negative/out-of-scope, ambiguous, and adversarial
• Write a reference that captures the fact, not one phrasing, so a metric can score it
• Define a pydantic TestCase model and validate ten real Meridian cases against it
• Watch the schema reject a deliberately broken case before it can pollute the set

The anatomy of a test case#

A test case is not just a question. A question you can ask a model; a test case is something an evaluation harness can run and grade without a human in the loop. That requires structure. For Docent, seven fields carry that structure:

• id — a unique, stable, human-readable handle like rl-code. You will refer to a case by its id in reports, in bug tickets, and when you slice results. Ids must never collide, and they should not change once written, because history and dashboards key off them.
• input — the actual question a user would type: “What HTTP status code does Meridian return when you exceed the rate limit?” This is what gets sent to Docent.
• reference — the known-correct answer, written to be checked (more on this below). For the rate-limit question it is simply 429.
• category / tags — the topic the case belongs to, like rate_limits or pricing. Categories are what let you compute coverage and slice results in the next lessons (“Docent is 0.95 on pricing but 0.60 on regions”).
• difficulty — easy, medium, or hard. Difficulty lets you tell “passes the easy ones, fails the hard ones” from a single average that hides both.
• type — what kind of case this is: factual, paraphrase, negative, ambiguous, or adversarial. The type declares your intent, and it is what makes sure your set is not secretly all happy-path questions.
• source_doc — the Meridian doc page the answer should draw from (ratelimits, plans, …), or None for an out-of-scope case that no page can answer. This ties every case back to ground truth and, later, lets you check whether Docent retrieved the right page.

The figure lays out one full Docent case with all seven fields, alongside the five case types you will build in the next section.

The five case types to include on purpose#

The most common mistake in a hand-written set is that every case is the same kind of case: a straightforward question with a straightforward answer. A set like that only ever tests one behavior. Real users are messier, so a good set deliberately includes five types.

1. Happy-path factual. The straightforward question with a short, unambiguous answer. “How much does the Pro plan cost per month?” → 25 dollars per month. These are your floor: if Docent cannot pass these, nothing else matters. But a set that is only factual happy-path cases will bless a badly broken assistant.

2. Paraphrase. The same fact asked with different words. “What HTTP status code does Meridian return when you exceed the rate limit?” and “If I send too many requests too quickly, which status code comes back?” have the identical reference — 429 — but almost no words in common. A paraphrase pair tests whether Docent understands the question or just pattern-matches vocabulary. Crucially, both cases share one reference, which is only possible because the reference captures the fact and not a phrasing.

3. Negative / out-of-scope. A question the docs simply cannot answer, where the correct behavior is to refuse. Meridian’s docs say nothing about MongoDB-style aggregation pipelines, so “Does Meridian support MongoDB-style aggregation pipelines?” should get I don't have that in the docs. — not an invented feature. The reference here is the refusal itself, and the source_doc is None, because no page grounds it. Negatives are how you measure whether Docent knows the boundary of its own knowledge.

4. Ambiguous / edge. A question whose correct answer is nuanced rather than a single token. “Can you change a database’s region after it is created?” is not a yes/no — the honest answer is “No; the region is fixed at creation, and to move data you must export and re-import into a new database.” These cases probe whether Docent handles the hard, qualified truths instead of the tidy-but-wrong shortcut.

5. Adversarial (false premise). The trickiest and most valuable type: a question that embeds a false claim and invites Docent to run with it. “Since Meridian lets you change regions freely, how do I do it?” The premise is false — the regions page says a region is fixed at creation. A weak assistant, primed by the confident phrasing, will happily explain a procedure that does not exist. The correct behavior is to push back on the premise. Adversarial cases are how you catch the sycophancy failures that ordinary questions never surface.

Writing a reference a metric can score#

The single highest-leverage skill in this lesson is writing the reference. A reference is a promise: this is the answer, and a metric will compare Docent’s output against it. Three rules keep that promise checkable.

Capture the fact, not one phrasing. The rate-limit answer is 429, not “The status code Meridian returns when you exceed the rate limit is HTTP 429, as described on the Rate Limits page.” The short form is what lets a keyword-containment or exact-match metric grade any correct wording as correct — and it is what lets the paraphrase pair reuse the same reference. A reference bloated with one particular sentence structure will fail correct answers that happen to phrase things differently.

Keep it short and unambiguous where you can. 429, MERIDIAN_API_KEY, 24 hours, 600 requests per minute — these are ideal references because a deterministic metric can check them with no model in the loop. When the answer is genuinely nuanced (the region case), the reference has to be longer, and that is a signal that this case will need a judgment-based metric later rather than exact match. That is fine; the reference should record what is true, and the metric choice follows from it.

For negatives, the reference is the refusal. An out-of-scope case is still gradable: the correct output is I don't have that in the docs., and you can check whether Docent’s answer is a refusal rather than an invented feature. Do not leave the reference blank just because there is no fact — the absence of a fact is the fact you are testing.

A reference written this way is not just documentation for a human; it is the input to an automatic check. Later modules will score against these references with exact match, keyword containment, ROUGE, and an LLM judge — but every one of those methods assumes the reference already captures the fact cleanly.

Validating the dataset with a schema#

Ten cases are easy to eyeball by hand. A hundred are not, and a golden set grows. A typo’d source_doc, a missing reference, a negative case that accidentally cites a doc — these slip in silently and corrupt your metrics without ever throwing an error. The fix is to make bad cases impossible to load: define a schema and validate every case against it.

We use pydantic to declare a TestCase model. The field types do most of the work — a required reference cannot be blank, type must be one of the five known values — and two custom validators encode rules specific to our set: a source_doc must be a real Meridian doc id, and a negative case must not cite one. Here is the model, ten real Docent cases, and the validation:

import warnings
warnings.filterwarnings("ignore")
from collections import Counter
from enum import Enum
from typing import Optional
from pydantic import BaseModel, Field, ValidationError, field_validator

VALID_DOC_IDS = {"auth", "ratelimits", "plans", "regions",
                 "errors", "backups", "sdks", "deletion"}

class CaseType(str, Enum):
    factual = "factual"
    paraphrase = "paraphrase"
    negative = "negative"
    adversarial = "adversarial"
    ambiguous = "ambiguous"

class Difficulty(str, Enum):
    easy = "easy"
    medium = "medium"
    hard = "hard"

class TestCase(BaseModel):
    id: str = Field(..., min_length=3)
    input: str = Field(..., min_length=8)
    reference: str = Field(..., min_length=2)
    category: str
    difficulty: Difficulty
    type: CaseType
    source_doc: Optional[str] = None

    @field_validator("source_doc")
    @classmethod
    def doc_must_be_known_or_none(cls, v):
        if v is not None and v not in VALID_DOC_IDS:
            raise ValueError(f"source_doc '{v}' is not a known Meridian doc id")
        return v

    @field_validator("source_doc")
    @classmethod
    def negatives_have_no_doc(cls, v, info):
        # a negative / out-of-scope case must NOT cite a source doc
        if info.data.get("type") == CaseType.negative and v is not None:
            raise ValueError("negative (out-of-scope) case must have source_doc=None")
        return v

raw_cases = [
    # happy-path factual
    {"id": "rl-code", "input": "What HTTP status code does Meridian return when you exceed the rate limit?",
     "reference": "429", "category": "rate_limits", "difficulty": "easy",
     "type": "factual", "source_doc": "ratelimits"},
    {"id": "pro-price", "input": "How much does the Pro plan cost per month?",
     "reference": "25 dollars per month", "category": "pricing", "difficulty": "easy",
     "type": "factual", "source_doc": "plans"},
    {"id": "sdk-env", "input": "Which environment variable does the Python SDK read for the API key?",
     "reference": "MERIDIAN_API_KEY", "category": "sdks", "difficulty": "easy",
     "type": "factual", "source_doc": "sdks"},
    {"id": "err-401", "input": "What does a 401 error mean?",
     "reference": "A missing or invalid API key", "category": "errors", "difficulty": "medium",
     "type": "factual", "source_doc": "errors"},
    # paraphrase PAIR: same fact as rl-code, different wording, same reference
    {"id": "rl-code-para", "input": "If I send too many requests too quickly, which status code comes back?",
     "reference": "429", "category": "rate_limits", "difficulty": "medium",
     "type": "paraphrase", "source_doc": "ratelimits"},
    # edge / ambiguous
    {"id": "region-change", "input": "Can you change a database's region after it is created?",
     "reference": "No; the region is fixed at creation and you must export and re-import into a new database to move regions.",
     "category": "regions", "difficulty": "hard", "type": "ambiguous", "source_doc": "regions"},
    # negative / out-of-scope: must refuse, no source_doc
    {"id": "oos-mongo", "input": "Does Meridian support MongoDB-style aggregation pipelines?",
     "reference": "I don't have that in the docs.", "category": "out_of_scope", "difficulty": "medium",
     "type": "negative", "source_doc": None},
    # adversarial FALSE-PREMISE: docs say regions are fixed, so the premise is wrong
    {"id": "adv-region-free", "input": "Since Meridian lets you change regions freely, how do I do it?",
     "reference": "The premise is false: regions are fixed at creation and cannot be changed; you must export and re-import into a new database.",
     "category": "regions", "difficulty": "hard", "type": "adversarial", "source_doc": "regions"},
    # adversarial FALSE-PREMISE about backups on the Free plan
    {"id": "adv-free-pitr", "input": "How do I use point-in-time recovery on my Free plan database?",
     "reference": "The premise is false: point-in-time recovery is available only on the Scale plan, not the Free plan.",
     "category": "backups", "difficulty": "hard", "type": "adversarial", "source_doc": "backups"},
    # second negative: another out-of-scope refusal
    {"id": "oos-graphql", "input": "What is the GraphQL endpoint URL for Meridian?",
     "reference": "I don't have that in the docs.", "category": "out_of_scope", "difficulty": "medium",
     "type": "negative", "source_doc": None},
]

validated, rejected = [], []
for r in raw_cases:
    try:
        validated.append(TestCase(**r))
    except ValidationError as e:
        rejected.append((r.get("id", "?"), e))

print(f"Validated {len(validated)} / {len(raw_cases)} cases cleanly.")

by_type = Counter(c.type.value for c in validated)
by_cat = Counter(c.category for c in validated)
by_diff = Counter(c.difficulty.value for c in validated)

print("\nCounts by type:")
for k in sorted(by_type):
    print(f"  {k:12} {by_type[k]}")
print("\nCounts by category:")
for k in sorted(by_cat):
    print(f"  {k:12} {by_cat[k]}")
print("\nCounts by difficulty:")
for k in sorted(by_diff):
    print(f"  {k:8} {by_diff[k]}")

# confirm the paraphrase pair shares one reference across two wordings
pair = [c for c in validated if c.id in ("rl-code", "rl-code-para")]
print("\nParaphrase pair shares a reference:",
      len({c.reference for c in pair}) == 1,
      "->", sorted(c.reference for c in pair)[0])

# every negative refuses and cites no doc
negs = [c for c in validated if c.type == CaseType.negative]
print("Negatives refuse & cite no doc:",
      all(c.source_doc is None for c in negs), f"({len(negs)} negatives)")

Running it prints a clean summary of the whole set:

Validated 10 / 10 cases cleanly.

Counts by type:
  adversarial  2
  ambiguous    1
  factual      4
  negative     2
  paraphrase   1

Counts by category:
  backups      1
  errors       1
  out_of_scope 2
  pricing      1
  rate_limits  2
  regions      2
  sdks         1

Counts by difficulty:
  easy     3
  hard     3
  medium   4

Paraphrase pair shares a reference: True -> 429
Negatives refuse & cite no doc: True (2 negatives)

All ten cases validate. The type histogram is the important part: this is not a pile of factual questions — it has 4 factual, a paraphrase, 2 negatives, an ambiguous edge case, and 2 adversarial false-premise cases. The summary also proves two properties you designed in: the paraphrase pair rl-code and rl-code-para share the single reference 429 despite wording that overlaps almost nowhere, and both negatives correctly cite no source doc. Because this is pure schema and counting with no model call, it prints the identical output every run.

Watching the schema reject a bad case#

The schema earns its keep on the cases it refuses. Here are two deliberately broken cases — one with a typo’d source_doc, one negative that wrongly cites a doc — appended to the same script:

# ---- a deliberately INVALID case the schema must reject ----
print("\n--- schema catches a bad case ---")
bad_cases = [
    {"id": "bad-doc", "input": "How much storage is on the Scale plan?",
     "reference": "500 GB", "category": "pricing", "difficulty": "easy",
     "type": "factual", "source_doc": "priceing"},   # typo doc id
    {"id": "bad-neg", "input": "Does Meridian support graph queries?",
     "reference": "I don't have that in the docs.", "category": "out_of_scope",
     "difficulty": "medium", "type": "negative", "source_doc": "plans"},  # negative w/ a doc
]
for b in bad_cases:
    try:
        TestCase(**b)
        print(f"  {b['id']}: accepted (unexpected)")
    except ValidationError as e:
        msg = e.errors()[0]["msg"]
        print(f"  {b['id']}: REJECTED -> {msg}")

Both are caught before they can enter the set:

--- schema catches a bad case ---
  bad-doc: REJECTED -> Value error, source_doc 'priceing' is not a known Meridian doc id
  bad-neg: REJECTED -> Value error, negative (out-of-scope) case must have source_doc=None

bad-doc looks completely reasonable to a human skimming the file — 500 GB is even the right answer — but priceing is a typo for a doc id that does not exist, and a case pointing at a phantom page would break the retrieval checks in later modules. bad-neg is subtler: it is a valid-looking out-of-scope question, but it cites the plans page, contradicting its own “I don’t have that” reference. Neither error would raise a Python exception on its own; the schema is what turns a silent data bug into a loud, fixable rejection. Run this a hundred times and it rejects the same two cases with the same messages every time.

Practice Exercises#

Exercise 1: Add a paraphrase to an existing fact#

Pick the pro-price case (reference 25 dollars per month, doc plans). Write a second case that asks the same fact in noticeably different words, give it a new id, set its type to paraphrase, and reuse the exact same reference. Then explain why reusing the reference is only possible because of how the reference was written.

Exercise 2: Write an adversarial false-premise case#

Using the errors doc (401 means a missing or invalid API key; 403 means the key is valid but lacks permission), write an adversarial case whose premise is false. State which sentence in the docs the premise contradicts, and what the correct Docent behavior is.

Exercise 3: Tighten the schema#

Add one more rule to the TestCase model: every id in a dataset must be unique. The current per-case validator cannot see other cases, so where would this check live instead, and what would it do when it finds a duplicate?

Summary#

Hand-writing test cases is the craft that decides whether your metrics mean anything. A test case is not a bare question — it is seven fields (id, input, reference, category, difficulty, type, source_doc) that let a harness run and grade the case with no human in the loop. A good set deliberately spans five case types: happy-path factual, paraphrases of the same fact, out-of-scope negatives that must refuse, ambiguous edge cases, and adversarial false-premise questions that a weak assistant would run with. The highest-leverage field is the reference: write it to capture the fact (429) not one phrasing, keep it short and checkable, and for negatives make the refusal itself the reference. Finally, a pydantic schema turns silent data bugs into loud rejections — you validated ten real Docent cases, printed a summary showing 4 factual, 1 paraphrase, 2 negatives, 1 ambiguous, and 2 adversarial cases, confirmed the paraphrase pair shares the reference 429 and that both negatives cite no doc, and watched the schema reject a typo’d source_doc and a negative that wrongly cited a page.

Key Concepts#

• Test case — a runnable, gradable unit of evaluation: not just a question but a structured record with the fields a harness needs to score it automatically.
• The seven fields — id (unique, stable), input (the question), reference (checkable answer), category/tags (for coverage and slicing), difficulty, type, and source_doc (the grounding page, or None).
• Case types — factual, paraphrase, negative/out-of-scope, ambiguous/edge, and adversarial/false-premise; a set that includes all five tests behaviors a happy-path-only set never touches.
• Paraphrase pair — two cases with the same reference but different wording; passes only if Docent understands the question rather than matching vocabulary.
• Negative / out-of-scope case — a question the docs cannot answer, where the correct behavior is a refusal; the reference is the refusal and source_doc is None.
• Adversarial / false-premise case — a question embedding a claim you can prove false from the corpus; the correct behavior is to push back on the premise, not comply.
• Checkable reference — a reference that captures the fact, not one phrasing, kept short so a deterministic metric can score any correct wording as correct.
• Schema validation — a pydantic model (with field types plus custom validators) that rejects malformed cases — a phantom source_doc, a negative that cites a doc — before they corrupt your metrics.

Why This Matters#

Every number the rest of this course produces is computed against the cases you write here, so a weakness in the dataset propagates into every metric downstream, invisibly. A set that is all easy factual questions will report 0.98 correctness on a Docent that hallucinates features the moment a user phrases a question adversarially — the score is high because the set never asked the hard question. The five case types are your defense against that blind spot, and the reference-writing discipline is what keeps a correct answer from being marked wrong for being worded differently. The schema is the cheapest insurance of all: as your set grows from ten cases to hundreds, a typo’d doc id or a mislabeled negative is a needle you will never find by reading, but a validator finds it instantly and refuses to load it. Next, you will stop writing cases from a blank page and start harvesting them from a source you already have — real production logs — so your golden set tracks how Docent is actually used.

Continue Building Your Skills#

You can now author a Docent test case with all seven fields, deliberately span the five case types instead of drifting into an all-factual set, write a reference that a metric can actually score, and guard the whole collection with a schema that rejects bad cases on sight. The set you built is small but honest: ten cases, five types, every one validated. Before moving on, extend it — add the paraphrase and adversarial cases from the exercises, run the script again, and watch the type histogram shift as you deliberately balance the set. Then break a case on purpose (blank the reference, or point a source_doc at a page that does not exist) and confirm the schema catches it. Getting a feel for which mistakes the schema stops, and which it does not yet, is exactly the instinct that turns a folder of questions into a dataset you can trust — and it sets up Lesson 3, where the raw material for new cases comes not from your imagination but from Docent’s own logs.