How to Evaluate LLM JSON Extraction Accuracy with Python

A large language model (LLM) is expected to return a JSON object for every customer request. The dashboard reports a 95% JSON parse rate, so the extractor appears dependable. Yet some outputs omit required fields, use the wrong type, or attach the wrong value to a valid field.

To evaluate LLM JSON extraction accuracy, test every saved output in three stages: parse the JSON, check its fields and types against a fixed contract, and compare each valid field with a reviewed reference record. Report parse rate, schema compliance, exact-record accuracy, and field accuracy separately, and count invalid outputs as failures in end-to-end accuracy.

This lesson builds that evaluator with standard Python and tests 20 saved outputs without making an API call. The example is deliberately small enough to inspect, and its failure rules are explicit enough to adapt to another extraction task.

Python Setup and Test Data

You need Python 3.10 or later and basic knowledge of dictionaries, loops, and functions. The evaluator uses only Python’s standard library. Matplotlib creates the result chart.

Create and activate a virtual environment, then install Matplotlib. On macOS or Linux, run:

python -m venv .venv
source .venv/bin/activate
python -m pip install matplotlib

On Windows PowerShell, activate the environment with .venv\Scripts\Activate.ps1. The executed build used Python 3.13.2 and Matplotlib 3.11.0.

Download the two input files and save them beside your Python program:

• reviewed delivery-request labels
• saved LLM outputs

You can also inspect the executed per-case results without running the evaluator.

The two input datasets are original synthetic teaching data released under CC0 1.0. Synthetic means the examples were created for this lesson. The request IDs, messages, parcel details, reference labels, and model-like mistakes are fictional. The outputs do not measure a real model.

Both inputs use JSON Lines, also called JSONL. Each line is a complete JSON object. The label file contains a request and its reviewed reference. The output file stores the raw model text inside an outer record, so one deliberately malformed response does not break the full file.

How Extraction Evaluation Works: Three Separate Gates

Structured extraction turns free text into named values that another program can use. A reference record is the reviewed answer for one test case. A schema is a contract that says which fields, types, and allowed values an output must have.

One output moves through three gates:

raw model text
  -> valid JSON syntax
  -> required shape, types, and allowed values
  -> field values equal the reviewed reference

The gates answer different questions. Parsing asks whether the text is valid JSON. Schema checking asks whether the parsed value has the structure the application expects. Reference comparison asks whether the extracted meaning is correct.

This distinction prevents a common reporting error. The array below is valid JSON, but it is not the object our application expects:

[{"intent":"cancel","parcel_count":3}]

An object can also follow the schema and still be wrong. For example, "parcel_count": 4 has a valid type and permitted range, but it fails the reference comparison if the reviewed request says five parcels. The structured JSON tutorial explains how to ask an API for a typed output. This tutorial starts after generation and measures whether saved outputs are usable and correct.

We use strict comparison because the schema already defines normalized values. intent must be one of four lowercase labels, and delivery_window must be morning, afternoon, evening, or null. If your task accepts equivalent forms such as a.m. and morning, define that normalization before labeling and apply it to both predictions and references.

Load Small Cases Before Calculating Metrics

Start with a helper that reads one JSON object from each non-empty line. Python’s json.loads() documentation says that the function converts JSON text to a Python value and raises JSONDecodeError when the document is invalid.

The next code loads the outer records, which are valid JSONL. It does not yet parse the nested model_output strings:

import json


def read_jsonl(path):
    with open(path, encoding="utf-8") as handle:
        return [
            json.loads(line)
            for line in handle
            if line.strip()
        ]


labels = read_jsonl("delivery_request_labels.jsonl")
saved_outputs = read_jsonl("saved_llm_outputs.jsonl")

print("label rows:", len(labels))
print("saved outputs:", len(saved_outputs))
print(labels[0]["request_text"])
print(saved_outputs[0]["model_output"])

The executed program printed:

label rows: 20
saved outputs: 20
Please move my one-parcel delivery to the morning. It does not require a signature.
{"intent":"reschedule","delivery_window":"morning","parcel_count":1,"needs_signature":false}

The first pair is easy to inspect. The request asks to reschedule one parcel for the morning and says no signature is needed. The output represents those four facts correctly. Do not calculate a score yet; first make the accepted output contract explicit.

Also confirm that IDs are unique and aligned. A duplicated or missing ID can compare an output with the wrong reference and produce a believable but false score:

references = {row["case_id"]: row["reference"] for row in labels}
output_ids = [row["case_id"] for row in saved_outputs]

assert len(references) == len(labels)
assert len(output_ids) == len(set(output_ids))
assert set(output_ids) == set(references)

These assertions have no printed output when they pass. If one fails, fix the evaluation data before measuring the extractor.

Check the Output Contract in Python

Our record has four fields. The intent and time window use closed lists of allowed values. Parcel count is an integer from 1 through 5, and needs_signature is a Boolean value.

Define the field names and allowed categories first. None is Python’s value for JSON null:

FIELDS = (
    "intent",
    "delivery_window",
    "parcel_count",
    "needs_signature",
)

ALLOWED_INTENTS = {
    "reschedule",
    "safe_place",
    "cancel",
    "delivery_question",
}
ALLOWED_WINDOWS = {"morning", "afternoon", "evening", None}

Now validate one parsed value. The function returns a short error instead of only False, because a useful evaluator must explain why a case stopped.

def validate_prediction(value):
    if not isinstance(value, dict):
        return "top level is not an object"

    keys = set(value)
    required = set(FIELDS)
    if keys != required:
        missing = sorted(required - keys)
        extra = sorted(keys - required)
        if missing:
            return "missing keys: " + ", ".join(missing)
        return "unexpected keys: " + ", ".join(extra)

    if value["intent"] not in ALLOWED_INTENTS:
        return "intent is outside the allowed values"
    if value["delivery_window"] not in ALLOWED_WINDOWS:
        return "delivery_window is outside the allowed values"

    count = value["parcel_count"]
    if type(count) is not int or not 1 <= count <= 5:
        return "parcel_count must be an integer from 1 through 5"
    if type(value["needs_signature"]) is not bool:
        return "needs_signature must be true or false"

    return None

The exact-key check rejects both missing and unexpected fields. This matches the closed contract used in the lesson. JSON Schema describes the same ideas with keywords such as required, type, enum, and additionalProperties; the official JSON Schema Draft 2020-12 page links the current core and validation specifications.

The expression type(count) is not int is intentional. In Python, bool is a subclass of int, so isinstance(True, int) returns True. A parcel count of true should still fail this contract.

For a larger schema, use a maintained validation library instead of expanding one long function. Keep the same evaluation stages: parser errors, schema errors, and value errors must remain distinguishable.

Score One Case Without Hiding Invalid Output

Next, combine parsing, validation, and field comparison. If parsing or schema checking fails, mark every extracted field as incorrect. This policy measures end-to-end usefulness: an application cannot safely consume a correct parcel count that appears inside an invalid record.

The following function returns one flat result row that can later become a CSV file:

def evaluate_case(case_id, raw_output, reference):
    try:
        parsed = json.loads(raw_output)
    except json.JSONDecodeError as exc:
        return {
            "case_id": case_id,
            "parse_ok": False,
            "schema_ok": False,
            "exact_record": False,
            **{f"{field}_correct": False for field in FIELDS},
            "error": f"invalid JSON at character {exc.pos}",
        }

    schema_error = validate_prediction(parsed)
    if schema_error:
        return {
            "case_id": case_id,
            "parse_ok": True,
            "schema_ok": False,
            "exact_record": False,
            **{f"{field}_correct": False for field in FIELDS},
            "error": schema_error,
        }

    field_results = {
        f"{field}_correct": parsed[field] == reference[field]
        for field in FIELDS
    }
    return {
        "case_id": case_id,
        "parse_ok": True,
        "schema_ok": True,
        "exact_record": all(field_results.values()),
        **field_results,
        "error": "" if all(field_results.values()) else "one or more values differ from the reference",
    }

There are two reasonable field metrics, and their denominators answer different questions:

• Conditional field accuracy uses only schema-valid records. It asks how often a field is correct after the output becomes usable.
• End-to-end field accuracy uses all evaluation cases. It asks how often the complete extraction pipeline delivers a usable correct field.

This tutorial reports end-to-end accuracy as the main result. Conditional accuracy can still help diagnosis, but it must be labeled clearly. Removing invalid outputs would raise the field scores while hiding failures that users and applications actually receive.

Run the Full Evaluation and Read the Real Output

Evaluate every saved output against the matching reference. The next loop preserves one result per case, including malformed and incomplete responses:

results = []

for saved in saved_outputs:
    case_id = saved["case_id"]
    results.append(
        evaluate_case(
            case_id,
            saved["model_output"],
            references[case_id],
        )
    )

total = len(results)
parse_count = sum(row["parse_ok"] for row in results)
schema_count = sum(row["schema_ok"] for row in results)
exact_count = sum(row["exact_record"] for row in results)

print(
    f"cases={total} parse_ok={parse_count} "
    f"schema_ok={schema_count} exact_records={exact_count}"
)

for field in FIELDS:
    correct = sum(row[f"{field}_correct"] for row in results)
    print(f"{field}: {correct}/{total} = {correct / total:.0%}")

The complete executed build produced:

cases=20 parse_ok=19 schema_ok=15 exact_records=10
intent: 13/20 = 65%
delivery_window: 13/20 = 65%
parcel_count: 14/20 = 70%
needs_signature: 14/20 = 70%

Read the first line as a narrowing sequence. Nineteen responses are valid JSON, but only 15 satisfy the output contract. Only 10 get all four values right. The 95% parse rate therefore does not justify calling the extractor 95% accurate.

The field rows include all 20 cases. Intent and delivery window are correct end to end in 13 cases. Parcel count and signature requirement are correct in 14. Exact-record accuracy is stricter: one wrong field makes the whole record wrong, so it is 10 divided by 20, or 50%.

The left panel shows where outputs stop. Four parsed values fail the schema because of a missing key, an extra key, a string where an integer is required, or an array where an object is required. Five more records satisfy the schema but contain at least one wrong value.

The right panel shows which fields need attention. It does not prove why they fail. Read the case rows before changing a prompt, model, or schema.

Print only the failed cases to make that review manageable:

for row in results:
    if not row["exact_record"]:
        print(row["case_id"], row["error"])

The executed failure list is:

D011 one or more values differ from the reference
D012 one or more values differ from the reference
D013 one or more values differ from the reference
D014 one or more values differ from the reference
D015 one or more values differ from the reference
D016 missing keys: needs_signature
D017 unexpected keys: confidence
D018 parcel_count must be an integer from 1 through 5
D019 top level is not an object
D020 invalid JSON at character 98

The published per-case result table adds a Boolean column for every field. That detail separates a broad schema problem from a narrow semantic error such as one wrong delivery window.

Apply the Evaluator to Your Own Extractor

Start from the decisions your application makes, not from a generic metric list. Write the output contract first. Define required fields, null behavior, allowed categories, numeric ranges, date formats, and whether extra fields are rejected.

Then build a reviewed evaluation set that includes normal cases and difficult cases from the intended task. Include missing information, explicit negation, several numbers in one input, optional values, long inputs, and inputs that should be rejected. Keep a stable ID for every case.

Run the extractor once and save its raw response before parsing. Also record the model name and version, prompt version, generation settings, schema version, and run date. Raw outputs let you rerun the evaluator after fixing a validator without paying for another model call or mixing model changes with evaluator changes.

Do not tune repeatedly on the final evaluation set. When you inspect a failure and change the prompt, that case has influenced development. Keep another reviewed set for the final comparison. If you compare two extractor versions on the same cases, the paired bootstrap tutorial shows how to report uncertainty around their difference.

For fields where not every value must be extracted, accuracy may be incomplete. Suppose the model can return a list of product codes from each message. You may also need precision for incorrect extra codes and recall for missed codes. Keep parse and schema failures in the evaluation report even when the semantic metric changes.

Common Mistakes and Troubleshooting

Only successfully parsed rows are scored. This creates selection bias because the cleanest outputs remain while operational failures disappear. Report both conditional and end-to-end scores, and use the end-to-end score for the full pipeline.

Valid JSON is treated as a valid record. JSON can contain an array, an unexpected field, or a string in place of a number. Validate the top-level shape, exact keys, types, ranges, and allowed values before comparison.

Schema compliance is treated as semantic accuracy. A schema accepts "intent": "cancel" when cancel is an allowed label. It cannot know that the request actually asked to reschedule. Compare with reviewed references after validation.

Missing output becomes an empty but valid value. Do not convert a parser error, refusal, timeout, or absent response into a record filled with defaults. Preserve the failure type. A default can look like a genuine model prediction.

Predictions and references are joined by row order. A skipped output shifts every later comparison. Join by a unique case ID and assert that the ID sets match.

Normalization is added after seeing failures. Decide acceptable equivalences before final scoring. A rule invented to rescue one prediction can hide a real error. Apply the same rule to both sides and test it separately.

The test set contains only easy positive wording. Add negation, omitted optional fields, more than one number, unfamiliar wording, and invalid requests. Use governed and de-identified real examples when your data policy permits them.

A synthetic score is presented as model performance. The values in this lesson verify the evaluator. They do not describe a vendor model. A real report must identify the tested system, dataset, labeling process, and evaluation date.

Recap

Structured extraction accuracy has three boundaries: JSON syntax, schema compliance, and semantic agreement with reviewed references. Measure each boundary separately so a high parse rate cannot hide unusable shapes or wrong values.

Keep every case in end-to-end field denominators. Preserve clear failure reasons, inspect exact-record failures, and version the prompt, schema, labels, and raw outputs. With those pieces in place, one score becomes a reviewable evaluation rather than a vague claim about quality.