Lesson 4 - Structured-Output & Schema Checks

Deterministic schema-check ladder
------------------------------------------------------------------
case                  pass   stage    detail
------------------------------------------------------------------
good, fenced          PASS   ok       valid
good, null doc_id     PASS   ok       valid
missing field         FAIL   schema   confidence: Field required
bad enum              FAIL   schema   doc_id: Value error, doc_id 'pricing' is not one of the 8 Meridian ids or null
confidence 1.5        FAIL   schema   confidence: Input should be less than or equal to 1
confidence as string  FAIL   schema   confidence: Input should be a valid number, unable to parse string as a number
not JSON at all       FAIL   parse    no JSON object found in the text
trailing prose        PASS   ok       valid
------------------------------------------------------------------
structured-output pass rate: 3/8 = 38%

run 2 -> pass rate: 3/8 = 38%

Live structured-output check over Docent (claude-haiku-4-5)
----------------------------------------------------------------------
question                                      pass   stage
----------------------------------------------------------------------
What HTTP status code does Meridian return ...PASS   ok
How much does the Pro plan cost per month?    PASS   ok
How long are backups retained on the Scale ...PASS   ok
Does Meridian support MongoDB-style aggrega...PASS   ok
----------------------------------------------------------------------
structured-output pass rate: 4/4 = 100%

```json
{
  "answer": "HTTP 429",
  "doc_id": "ratelimits",
  "confidence": 0.95
}

The model wrapped its JSON in a ```` ```json ```` fence -- exactly the rung-1 hazard from the ladder. A naive `json.loads(raw)` on that string would have thrown, and a naive eval would have scored this *correct* reply as a format failure. Because `extract_json` strips the fence first, the reply is graded on its content and passes. That one detail is the difference between a live pass rate of 100% and a spuriously low one: the model was fine; only the wrapping was in the way. This is why you build the fence-stripping *before* you trust the pass rate -- and why a low structured-output score should always send you to look at raw replies, not straight to blaming the model.

Live outputs will shift from run to run: a different phrasing of `answer`, a `confidence` of `0.9` instead of `0.95`, occasionally a stray sentence outside the JSON. The validator's job is to give an honest verdict on whatever comes back, every time -- and to tell you, via the `stage`, whether a miss was a parse problem you can engineer around or a genuine schema violation the model produced.

---

## Practice Exercises

### Exercise 1: Add the cross-field (rung 5) constraint

The `DocentReply` model covers rungs 2 through 4, but not rung 5. Add a check that enforces the constraint *if `doc_id` is `null`, the `answer` must look like a refusal* (and, conversely, a non-null `doc_id` should not be a refusal). Then feed it an incoherent-but-well-typed reply like `{"answer": "The Pro plan costs 25 dollars per month.", "doc_id": null, "confidence": 0.95}` and confirm it now fails at the schema stage.


  
  

    

        
Hint

      
    

      
Cross-field rules need to see the whole object, so use a @model_validator(mode="after") rather than a single-field validator. Inside it, define a small refusal test – e.g. is_refusal = "don't have" in self.answer.lower() or "not in the docs" in self.answer.lower() – and raise ValueError("doc_id is null but answer is not a refusal") when self.doc_id is None and not is_refusal. Keep the refusal test simple here; a later module builds a proper refusal classifier instead of a substring check.

    
  



### Exercise 2: Break out the pass rate by rung

A single pass rate hides *where* replies fail. Modify the deterministic loop over `CASES` to also count how many failures happened at each stage (`parse` vs `schema`), and print those counts. On the eight hand-crafted cases, how many fail at `parse` and how many at `schema`? Explain why that split would change your fix if it came from live traffic instead.


  
  

    

        
Hint

      
    

      
A collections.Counter over the stage returned by validate for the failing cases gives the breakdown in two lines: stages = Counter(validate(raw)[1] for _, raw in CASES if not validate(raw)[0]). On CASES you’ll get one parse failure (the not JSON at all string) and four schema failures. In production, a spike in parse failures points at wrapping/formatting you can fix with extract_json or JSON mode; a spike in schema failures points at the model inventing fields, enum values, or out-of-range numbers – a prompt or model problem, not a parsing one.

    
  



### Exercise 3: Make Docent stumble, then catch it

The live prompt asks nicely for "a single JSON object and nothing else." Write a *deliberately weaker* prompt -- for instance, "Answer as JSON with an answer, the doc id, and your confidence," with no instruction to omit surrounding prose -- run it live on the four questions, and print the raw replies alongside the `validate` verdict. Does the weaker prompt produce more fenced blocks or trailing prose? Does `extract_json` still recover them, and does the pass rate hold?


  
  

    

        
Hint

      
    

      
Copy docent_json to docent_json_loose and swap only the instruction text; keep max_tokens small and reuse the same four questions to keep the call cost tiny. The interesting outcome isn’t a number, it’s the raw text: weaker prompts tend to add more prose and fences, which is exactly what extract_json is built to survive – so a well-built validator often keeps the pass rate high even when the prompt gets sloppy. If a reply ever fails, print its stage to see whether it was the prose (rung 1) or a genuine schema slip (rungs 2-4). Remember the answer varies run to run, so report what you actually saw, not what you expected.

    
  



---

## Summary

Structured output is its own category of evaluation, and one teams underrate: whenever an LLM feature emits machine-readable JSON, "correct" begins with "parses and validates," because a malformed reply breaks the program that consumes it before anyone asks whether the content was right. You grade it with a five-rung check ladder -- (1) parses as JSON, (2) required fields present, (3) correct types, (4) values in the allowed set or range, (5) cross-field constraints -- where each rung is a deterministic pass/fail and the rung that catches a reply names the defect. You built a `pydantic` validator for Docent's `{answer, doc_id, confidence}` contract, with `doc_id` constrained to Meridian's 8 pages or `null` and `confidence` in \([0, 1]\), plus a robust `extract_json()` that strips markdown fences and surrounding prose so a wrapped-but-valid reply is graded fairly. A deterministic run over eight hand-crafted strings gave a reproducible 3/8 pass rate -- catching a missing field, a bad enum, an out-of-range and a mistyped `confidence`, and plain non-JSON, while correctly passing two cosmetically-wrapped good cases. Live on four Docent questions the pass rate was 4/4, and a verbatim raw reply showed exactly why `extract_json` matters: the model fenced its JSON, which a naive parser would have failed.

### Key Concepts
- **Structured-output eval** -- grading outputs by *shape* (a schema) rather than by free-text quality; "correct" starts with "parses and validates."
- **The check ladder** -- parse → required fields → types → enums/ranges → cross-field constraints; an ordered sequence of deterministic gates, each catching a distinct failure.
- **Structured-output pass rate** -- the fraction of replies that clear all rungs; report it alongside the *distribution of failures by rung*, which tells you what to fix.
- **`extract_json`** -- a robust parse step that strips markdown fences and surrounding prose, so a structurally-valid reply isn't failed for cosmetic wrapping.
- **Schema as reference** -- a `pydantic` model (or JSON Schema) encodes required keys, types, enums, and bounds; validation is fully reproducible, unlike the model output it grades.
- **Necessary, not sufficient** -- passing the schema means *usable*, not *accurate*; pair the pass rate with content metrics (exact match, F1) to judge correctness.

### Why This Matters
Structured output is everywhere in production LLM systems -- routers, extractors, tool calls, function arguments, any feature whose result another service reads -- and a malformed reply there isn't a cosmetic blemish, it's an outage or a silent data-corruption bug. Yet format validity is the single most measurable thing about an LLM feature: a few lines of deterministic code give you a pass rate you can track on every deploy, alert on when it drops, and break down by rung to know whether the fix is a parser tweak or a prompt change. Teams that skip this ship "JSON mode" features whose real parse rate they've never measured and discover the gap in an incident. Teams that build the ladder catch a fenced-JSON regression in an eval run instead. You now have a validator that turns "does the output follow the structure?" into a number -- and the next lesson assembles this, exact match, F1, and ROUGE into a single reference-metric scorecard for Docent.

---

## Continue Building Your Skills

You can now grade an output nobody reads as prose -- the JSON another program consumes -- by walking it up a five-rung ladder and reporting a structured-output pass rate, with `extract_json` making the check fair to a model that likes to wrap its answer in fences. That's a deterministic, reproducible gate you can run on every deploy: 3/8 on the hand-crafted cases that must fail for specific reasons, 4/4 live on Docent when the wrapping is handled. The next lesson brings the whole module together. In the guided project you'll fold this schema check in with exact match, token-level F1, and ROUGE to build a single reference-metric scorecard -- one function that takes Docent's answers and a golden set and returns every deterministic number you've learned, in a form you can diff between two versions of the assistant.