Lesson 4 - Coverage, Slicing & Dataset Hygiene

Welcome to Coverage, Slicing & Dataset Hygiene#

In Lessons 2 and 3 you built Docent’s golden set by hand and by mining production logs. You now have a pile of tagged test cases for our Meridian documentation assistant. But a pile of cases is not the same as a trustworthy dataset, and the gap between those two is exactly what this lesson closes.

Three questions decide whether you can trust an eval set. First, coverage: does the set actually exercise everything that matters — every Meridian doc page, every difficulty, every question type, including the adversarial ones? Coverage is what turns the vague claim “we tested Docent” into the specific claim “we tested these things,” and it makes the untested corners visible. Second, slicing: a single 90% average can be 100% on easy questions and 33% on the hard ones, and that average will happily hide the failure until a user hits it. Reporting accuracy per subgroup is the single most important habit for honest evaluation. Third, hygiene: no duplicates inflating one category, no held-out test question leaking into the data you tune on, a version stamp so you know what “the golden set” even means today.

You will build all three as small, deterministic Python that runs the same twice — then run Docent live against the hard slice to see whether the current build really handles it.

In this lesson, you will:

• Compute a coverage report over Docent’s test set and find a Meridian doc page no question touches
• Confirm the set contains hard and adversarial cases, not just easy factoids
• Slice accuracy by difficulty and question type to expose a weak subgroup an average hides
• Run hygiene checks that flag a near-duplicate question and a held-out question leaking into new candidates
• Run Docent live on the hard slice and reconcile the real result with what slicing told you to watch

The test set, tagged for coverage#

Everything in this lesson runs on one dataset: Docent’s golden set, where each case carries the tags that make coverage and slicing possible. A bare question/reference pair tells you nothing about what it exercises; the tags do. Each case records its source_doc (which of the 8 Meridian pages it comes from), a category, a difficulty, and a qtype (factoid, reasoning, or adversarial).

Here is the set — the canonical golden questions from Module 2, now fully tagged — together with the list of all 8 Meridian doc pages Docent can draw on:

from collections import Counter

# The 8 Meridian doc pages that Docent can draw on.
DOC_PAGES = ["auth", "ratelimits", "plans", "regions",
             "errors", "backups", "sdks", "deletion"]

# Docent's golden test set, each case tagged for coverage and slicing.
# category / difficulty / source_doc / qtype describe WHAT each case exercises.
GOLDEN = [
    {"id": "g01", "q": "What HTTP status code does Meridian return when you exceed the rate limit?",
     "ref": "429", "source_doc": "ratelimits", "category": "rate-limits",
     "difficulty": "easy", "qtype": "factoid"},
    {"id": "g02", "q": "How many requests per minute does the Pro plan allow?",
     "ref": "600 requests per minute", "source_doc": "ratelimits", "category": "rate-limits",
     "difficulty": "easy", "qtype": "factoid"},
    {"id": "g03", "q": "How much does the Pro plan cost per month?",
     "ref": "25 dollars per month", "source_doc": "plans", "category": "pricing",
     "difficulty": "easy", "qtype": "factoid"},
    {"id": "g04", "q": "Can you change a database's region after it is created?",
     "ref": "No; the region is fixed at creation and you must export and re-import into a new database.",
     "source_doc": "regions", "category": "regions", "difficulty": "hard", "qtype": "reasoning"},
    {"id": "g05", "q": "How long are backups retained on the Scale plan?",
     "ref": "90 days", "source_doc": "backups", "category": "backups",
     "difficulty": "medium", "qtype": "factoid"},
    {"id": "g06", "q": "Which environment variable does the Python SDK read for the API key?",
     "ref": "MERIDIAN_API_KEY", "source_doc": "sdks", "category": "sdks",
     "difficulty": "medium", "qtype": "factoid"},
    {"id": "g07", "q": "What does a 401 error mean?",
     "ref": "A missing or invalid API key", "source_doc": "errors", "category": "errors",
     "difficulty": "easy", "qtype": "factoid"},
    {"id": "g08", "q": "How long is the grace period before a deleted database is permanently purged?",
     "ref": "24 hours", "source_doc": "deletion", "category": "deletion",
     "difficulty": "medium", "qtype": "factoid"},
    {"id": "g09", "q": "Which plans support point-in-time recovery?",
     "ref": "Only the Scale plan", "source_doc": "backups", "category": "backups",
     "difficulty": "hard", "qtype": "reasoning"},
    {"id": "g10", "q": "What does Meridian charge for the Free plan?",
     "ref": "0 dollars per month, including 1 GB of storage", "source_doc": "plans",
     "category": "pricing", "difficulty": "easy", "qtype": "factoid"},
    {"id": "g11", "q": "Does Meridian support MongoDB-style aggregation pipelines?",
     "ref": "(not in the docs; Docent should decline)", "source_doc": None,
     "category": "out-of-scope", "difficulty": "hard", "qtype": "adversarial"},
]

Notice case g11 has source_doc: None — it is the out-of-scope probe, whose correct answer is a refusal. Those tags are the raw material for every measurement below.

Coverage: what did we actually test?#

The first honest question about any test set is embarrassingly simple: did it touch everything? Docent answers from 8 Meridian doc pages. If no question in your set comes from the auth page, then every green checkmark in your report is silent about authentication — you did not test it, you just did not notice that you did not test it. Coverage makes that silence audible.

A coverage report is nothing more than counting. For each dimension you care about — doc page, category, difficulty, question type — tally how many cases land in each bucket, and flag any bucket that is empty when it should not be. The function below computes all of that and, crucially, lists which of the 8 doc pages have no question pointing at them:

def coverage_report(cases, doc_pages):
    covered = {c["source_doc"] for c in cases if c["source_doc"]}
    missing = [p for p in doc_pages if p not in covered]
    return {
        "n_cases": len(cases),
        "docs_covered": sorted(covered),
        "docs_missing": missing,
        "by_category": dict(sorted(Counter(c["category"] for c in cases).items())),
        "by_difficulty": dict(sorted(Counter(c["difficulty"] for c in cases).items())),
        "by_qtype": dict(sorted(Counter(c["qtype"] for c in cases).items())),
        "has_adversarial": any(c["qtype"] == "adversarial" for c in cases),
        "has_hard": any(c["difficulty"] == "hard" for c in cases),
    }

rep = coverage_report(GOLDEN, DOC_PAGES)
print(f"Test cases:        {rep['n_cases']}")
print(f"Doc pages covered: {len(rep['docs_covered'])}/{len(DOC_PAGES)}  {rep['docs_covered']}")
print(f"Doc pages MISSING: {rep['docs_missing']}")
print(f"By category:       {rep['by_category']}")
print(f"By difficulty:     {rep['by_difficulty']}")
print(f"By question type:  {rep['by_qtype']}")
print(f"Has adversarial?   {rep['has_adversarial']}")
print(f"Has hard cases?    {rep['has_hard']}")

This is pure counting with no model in the loop, so it is fully deterministic. Running it twice gives the identical report both times:

Test cases:        11
Doc pages covered: 7/8  ['backups', 'deletion', 'errors', 'plans', 'ratelimits', 'regions', 'sdks']
Doc pages MISSING: ['auth']
By category:       {'backups': 2, 'deletion': 1, 'errors': 1, 'out-of-scope': 1, 'pricing': 2, 'rate-limits': 2, 'regions': 1, 'sdks': 1}
By difficulty:     {'easy': 5, 'hard': 3, 'medium': 3}
By question type:  {'adversarial': 1, 'factoid': 8, 'reasoning': 2}
Has adversarial?   True
Has hard cases?    True

Read the report top to bottom and it tells a real story. Seven of the eight doc pages are covered, but auth is not — the golden set has zero questions about API keys, bearer tokens, or test-versus-live mode, so Docent’s behavior on authentication is currently untested. The difficulty spread is reasonable (5 easy, 3 medium, 3 hard) and there is one adversarial case, so the set is not all easy factoids. But the question-type tally is lopsided: 8 factoids to only 2 reasoning and 1 adversarial. That is a to-do list, not a disaster — write a couple of auth questions, add a few reasoning and adversarial cases — and you only have a to-do list because you counted.

Slicing: the average is a liar#

Coverage tells you the set touches the hard cases and the adversarial probe. Slicing tells you how Docent does on them — separately, so a strong majority cannot drown out a weak minority.

Here is the trap in one sentence: an average is a weighted vote, and the easy cases usually win. If 8 of your 11 cases are easy factoids and Docent aces them, the overall number will look great even if it fails most of the hard ones, because there simply are not enough hard cases to move the mean. The fix is to stop reporting one number and start reporting one number per subgroup.

To demonstrate this with reproducible arithmetic, we use a fixed, recorded list of per-item pass/fail results from an earlier Docent evaluation run — real-shaped labels, held constant so the slicing math is identical every time. (Further down, we run Docent live for the real, current picture.)

from collections import defaultdict

# Illustrative per-item results from a PRIOR Docent evaluation run.
# passed = did Docent's answer match the reference (or correctly refuse)?
# These are recorded, fixed numbers so the slicing math below is reproducible.
RESULTS = [
    {"id": "g01", "difficulty": "easy",   "qtype": "factoid",     "passed": True},
    {"id": "g02", "difficulty": "easy",   "qtype": "factoid",     "passed": True},
    {"id": "g03", "difficulty": "easy",   "qtype": "factoid",     "passed": True},
    {"id": "g04", "difficulty": "hard",   "qtype": "reasoning",   "passed": False},
    {"id": "g05", "difficulty": "medium", "qtype": "factoid",     "passed": True},
    {"id": "g06", "difficulty": "medium", "qtype": "factoid",     "passed": True},
    {"id": "g07", "difficulty": "easy",   "qtype": "factoid",     "passed": True},
    {"id": "g08", "difficulty": "medium", "qtype": "factoid",     "passed": True},
    {"id": "g09", "difficulty": "hard",   "qtype": "reasoning",   "passed": True},
    {"id": "g10", "difficulty": "easy",   "qtype": "factoid",     "passed": True},
    {"id": "g11", "difficulty": "hard",   "qtype": "adversarial", "passed": False},
]

def accuracy(results):
    return sum(r["passed"] for r in results) / len(results)

def sliced_accuracy(results, key):
    buckets = defaultdict(list)
    for r in results:
        buckets[r[key]].append(r["passed"])
    return {k: (sum(v) / len(v), len(v)) for k, v in buckets.items()}

overall = accuracy(RESULTS)
print(f"OVERALL accuracy: {overall:.1%}  ({sum(r['passed'] for r in RESULTS)}/{len(RESULTS)})")
print()
for key in ["difficulty", "qtype"]:
    print(f"Sliced by {key}:")
    for slice_name, (acc, n) in sorted(sliced_accuracy(RESULTS, key).items()):
        flag = "  <-- weak slice" if acc < 0.6 else ""
        print(f"  {slice_name:12} {acc:6.1%}  (n={n}){flag}")
    print()

The math is deterministic; run it twice and the numbers are identical:

OVERALL accuracy: 81.8%  (9/11)

Sliced by difficulty:
  easy         100.0%  (n=5)
  hard          33.3%  (n=3)  <-- weak slice
  medium       100.0%  (n=3)

Sliced by qtype:
  adversarial    0.0%  (n=1)  <-- weak slice
  factoid      100.0%  (n=8)
  reasoning     50.0%  (n=2)  <-- weak slice

Look at what the overall number was hiding. 81.8% sounds like a solid, ship-it grade. But slice by difficulty and the truth appears: Docent is perfect on easy and medium questions and gets only 33.3% of the hard ones right. Slice by question type and it is starker still — every factoid passes, but the reasoning cases are a coin flip and the single adversarial probe fails outright: Docent invented an answer to the MongoDB question instead of refusing. The exact failures that hurt a documentation assistant most — the tricky reasoning and the “please refuse this” cases — are precisely the ones the average smoothed over.

The figure below is that story in one picture: three tall bars and one short red one, with the overall average drawn as a dashed line that sits comfortably above the slice that is actually failing.

Does the current build really fail the hard slice? Run it live.#

The slice table above came from a fixed, recorded results list, chosen so the arithmetic is reproducible and the weak slice is unmistakable. That is the right way to teach slicing, but you should never confuse recorded numbers with the state of your app today. So let us do the honest thing and run Docent live against the three hard cases, grading each with a small keyword check — containment for the reasoning questions, a refusal check for the adversarial one:

import warnings
warnings.filterwarnings("ignore")
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

DOCS = [
    {"id": "auth", "title": "Authentication",
     "text": "Meridian authenticates every request with an API key sent in the Authorization header as a bearer token. "
             "Create keys in the dashboard under Settings > API Keys. Keys are either test or live mode; test keys only "
             "touch sandbox data. A key is shown once at creation and cannot be recovered, only rotated."},
    {"id": "ratelimits", "title": "Rate Limits",
     "text": "The Free plan allows 60 requests per minute. The Pro plan allows 600 requests per minute. Exceeding the "
             "limit returns HTTP 429 with a Retry-After header giving the seconds to wait. Rate limits are applied per "
             "API key, not per account."},
    {"id": "plans", "title": "Plans and Pricing",
     "text": "Meridian has three plans. Free costs 0 dollars per month and includes 1 GB of storage. Pro costs 25 dollars "
             "per month and includes 50 GB of storage. Scale costs 199 dollars per month and includes 500 GB of storage. "
             "All plans include unlimited read replicas."},
    {"id": "regions", "title": "Regions",
     "text": "Databases can be created in three regions: us-east, eu-west, and ap-south. The region is fixed at creation "
             "time and cannot be changed later; to move data to another region you must export and re-import into a new "
             "database. Cross-region read replicas are available on the Scale plan only."},
    {"id": "backups", "title": "Backups",
     "text": "Automatic daily backups are retained for 7 days on the Free plan, 30 days on Pro, and 90 days on Scale. "
             "Backups run at 03:00 UTC. Point-in-time recovery to any second in the retention window is available on the "
             "Scale plan. Restoring a backup creates a new database and never overwrites the original."},
]

def retrieve(question, docs, k=2):
    q = set(question.lower().split())
    scored = sorted(docs, key=lambda d: len(q & set((d["title"] + " " + d["text"]).lower().split())), reverse=True)
    return scored[:k]

def docent_answer(question, docs, model="claude-haiku-4-5"):
    ctx = "\n\n".join(f"[{d['id']}] {d['title']}\n{d['text']}" for d in retrieve(question, docs))
    prompt = (f"Answer the question using ONLY the context. If the context does not contain the answer, say "
              f"\"I don't have that in the docs.\" Be concise.\n\nContext:\n{ctx}\n\nQuestion: {question}")
    msg = client.messages.create(model=model, max_tokens=200,
                                 messages=[{"role": "user", "content": prompt}])
    return msg.content[0].text

# The three HARD-slice cases, with a simple keyword grader for each.
HARD = [
    {"id": "g04", "q": "Can you change a database's region after it is created?",
     "must_contain": ["export"]},          # correct method: export and re-import
    {"id": "g09", "q": "Which plans support point-in-time recovery?",
     "must_contain": ["scale"]},           # only the Scale plan
    {"id": "g11", "q": "Does Meridian support MongoDB-style aggregation pipelines?",
     "refuse": True},                       # out of scope: must decline
]

REFUSAL = ["don't have", "do not have", "not in the docs", "no information", "isn't in", "not covered"]

def grade(ans, case):
    a = ans.lower()
    if case.get("refuse"):
        return any(p in a for p in REFUSAL)
    return all(kw in a for kw in case["must_contain"])

passes = 0
for c in HARD:
    ans = docent_answer(c["q"], DOCS)
    ok = grade(ans, c)
    passes += ok
    print(f"{c['id']}  passed={ok}")
    print(f"   Q: {c['q']}")
    print(f"   A: {ans}")
print(f"\nHard-slice accuracy (live): {passes}/{len(HARD)} = {passes/len(HARD):.1%}")

Because this calls the live model, the wording varies run to run — here is one real recorded run:

g04  passed=True
   Q: Can you change a database's region after it is created?
   A: No. According to the context, the region is fixed at creation time and cannot be changed later. To move data to another region, you must export and re-import into a new database.
g09  passed=True
   Q: Which plans support point-in-time recovery?
   A: According to the context, only the **Scale plan** supports point-in-time recovery. It allows recovery to any second within the retention window (90 days for Scale).
g11  passed=True
   Q: Does Meridian support MongoDB-style aggregation pipelines?
   A: I don't have that in the docs.

Hard-slice accuracy (live): 3/3 = 100.0%

Good news: the current Docent build scores 3/3 on the hard slice — it reasons through the region and point-in-time-recovery questions and correctly refuses the MongoDB probe. That is not a contradiction of the earlier 33.3%; it is the whole point. The recorded results showed you what a weak hard slice looks like and why an average would bury it. The live run shows this build is not that build. And you can only make that claim — “our hard slice is healthy right now” — because you sliced and checked. Drop slicing and the day a model update quietly breaks refusals, your 82% overall will barely twitch and nobody will notice until a user does. Slicing is the instrument; the reading changes with every build, which is exactly why you keep taking it.

Hygiene: keep the set honest over time#

A covered, sliced dataset can still rot. Three kinds of rot matter most, and two of them you can catch with a few lines of code.

Duplicates quietly reweight your average. If “How much does the Pro plan cost per month?” appears twice under slightly different wording, pricing now counts double, and Docent’s pricing performance tugs the overall number harder than you intended. Leakage is worse: if a question you hold out to measure Docent also shows up in the data you use to tune Docent’s prompt, you are grading the exam with the answer key — the score is inflated and meaningless. The check below screens a batch of candidate questions before they enter the set: it flags near-duplicate pairs by word overlap (Jaccard similarity) and flags any candidate that collides with a held-out test question.

from itertools import combinations

# A candidate batch of questions to ADD to the golden set. Two of them are
# near-duplicates of each other, and one leaks a held-out test question.
CANDIDATES = [
    {"id": "c01", "q": "How much does the Pro plan cost per month?"},
    {"id": "c02", "q": "How much does the Pro plan cost each month?"},   # near-dup of c01
    {"id": "c03", "q": "Which region can a database be moved to after creation?"},
    {"id": "c04", "q": "How long is the grace period before a deleted database is permanently purged?"},  # leaks g08
]

# The held-out cases we must NOT tune on (a slice of the golden set).
HELD_OUT = [
    {"id": "g08", "q": "How long is the grace period before a deleted database is permanently purged?"},
]

STOP = {"the", "a", "an", "is", "of", "to", "does", "do", "what", "which",
        "how", "can", "be", "after", "for", "per"}

def tokens(text):
    return {w.strip("?.,") for w in text.lower().split()} - STOP

def jaccard(a, b):
    ta, tb = tokens(a), tokens(b)
    return len(ta & tb) / len(ta | tb) if (ta | tb) else 0.0

# 1) Deduplication: flag candidate pairs that are near-identical in meaning.
print("Near-duplicate check (Jaccard >= 0.60):")
for x, y in combinations(CANDIDATES, 2):
    j = jaccard(x["q"], y["q"])
    if j >= 0.60:
        print(f"  {x['id']} ~ {y['id']}  score={j:.2f}  DUPLICATE")

# 2) Leakage: flag any candidate that matches a held-out test question.
print("\nLeakage check against held-out set (Jaccard >= 0.90):")
for c in CANDIDATES:
    for h in HELD_OUT:
        j = jaccard(c["q"], h["q"])
        if j >= 0.90:
            print(f"  {c['id']} leaks held-out {h['id']}  score={j:.2f}  DROP")

Word-overlap is deterministic, so this returns the same verdicts every run:

Near-duplicate check (Jaccard >= 0.60):
  c01 ~ c02  score=0.83  DUPLICATE

Leakage check against held-out set (Jaccard >= 0.90):
  c04 leaks held-out g08  score=1.00  DROP

The check caught both problems. c01 and c02 are the same pricing question with “per month” swapped for “each month” — an 0.83 overlap, well past the duplicate threshold, so you keep one and drop the other before pricing gets double-counted. And c04 is a word-for-word copy of held-out case g08; letting it into the tuning pool would leak your test answer, so it is flagged to drop. This is a deliberately simple lexical check — real pipelines often add embedding similarity to catch reworded duplicates that share few words — but even this much stops the two most common ways a dataset silently corrupts itself.

The third kind of rot has no one-liner: staleness. When Meridian’s docs change — say the Pro plan’s price moves from 25 to 30 dollars — every golden case whose reference encodes the old fact becomes wrong, and Docent gets penalized for being right. The defense is process, not code: version the dataset (a version field and a changelog, so “the golden set” always means a specific, dated thing), and re-check labels whenever the source docs change. A test set is a living artifact; treat a docs update as a trigger to revisit the cases it touches.

Practice Exercises#

Exercise 1: Close the coverage gap#

The coverage report flagged auth as the one Meridian doc page with no question. Write a single new golden case for it — a factoid with a short, checkable reference drawn from the auth page text — and give it the same tag fields the other cases use (source_doc, category, difficulty, qtype). Then state what docs_missing would become after you add it.

Exercise 2: Slice by document#

The lesson sliced by difficulty and qtype. Using the same sliced_accuracy function and the recorded RESULTS, slice by a third dimension — source_doc — after joining each result to its case’s source_doc. Which document page would show the weakest accuracy, and why is that slice worth watching even though only a few cases fall in it?

Exercise 3: Tighten the duplicate threshold#

The near-duplicate check uses a Jaccard threshold of 0.60 and flagged c01 ~ c02 at 0.83. Suppose you add a candidate c05 = "What is the Pro plan monthly cost?". By hand or in code, estimate its Jaccard overlap with c01 under the lesson’s tokens function, and decide whether a 0.60 threshold would catch it. What does your answer say about choosing the threshold?

Summary#

A golden set becomes trustworthy only after you measure it, and this lesson built the three measurements that do it — all on Docent’s Meridian test set, all deterministic where it counts. Coverage was pure counting: it revealed that 7 of 8 doc pages are tested but auth is a blind spot, confirmed the set holds hard and adversarial cases, and exposed a factoid-heavy question mix — a to-do list you only get by counting. Slicing was the headline habit: an 81.8% overall average, sliced by difficulty, fell to 33.3% on the hard cases and 0% on the single adversarial probe, proving that one number is a weighted vote the easy cases win. Running Docent live on the hard slice then scored 3/3, a reminder that slices read differently per build and that you can only claim a healthy hard slice because you keep measuring it. Hygiene caught a near-duplicate pricing question (Jaccard 0.83) and a held-out question leaking into new candidates (Jaccard 1.00), and pointed at versioning as the defense against stale labels when the docs change.

Key Concepts#

• Coverage — the share of the things that matter (doc pages, categories, difficulties, question types) that your test set actually exercises; an empty bucket is an untested blind spot, found by counting.
• Coverage gap — a bucket with zero cases, like Docent’s untested auth page; it makes every other score silent about that area.
• Slicing — reporting accuracy per subgroup instead of one average, so a strong majority (easy factoids) cannot hide a failing minority (hard or adversarial cases).
• Weak slice — a subgroup whose accuracy is far below the overall number; here, hard cases at 33.3% under an 81.8% average.
• Deduplication — removing near-identical cases (flagged by word-overlap or embedding similarity) so no topic is silently double-weighted in the average.
• Leakage — a held-out test case appearing in the data you tune on; it inflates the score to meaninglessness, so held-out questions must be screened out of new candidates.
• Versioning / staleness — stamping the dataset with a version and revisiting labels when the source docs change, so results are comparable and references stay correct.

Why This Matters#

The difference between “we tested Docent” and “we tested Docent’s hard reasoning cases, its refusals, and all eight doc pages, and here is the accuracy on each” is the difference between a number that reassures you and a number you can act on. Coverage tells you what your evaluation can even see; slicing keeps a strong average from hiding the exact failures — bad reasoning, missing refusals — that erode user trust fastest; hygiene keeps the whole thing from silently corrupting as the set and the docs evolve. Teams that skip these ship on a comfortable 82% and get blindsided by the 33% slice in production. In the next lesson you will put all of it together, assembling Docent’s golden dataset as a single versioned, tagged, covered, and hygiene-checked artifact that every metric in the modules ahead will run against.

Continue Building Your Skills#

You can now interrogate an eval set the way a skeptic would: does it cover every doc page and question type, does any subgroup fail behind a healthy-looking average, and is it free of duplicates and leakage? You proved each check on Docent’s Meridian set — a missing auth page, an 81.8% average hiding a 33.3% hard slice, a flagged duplicate and a flagged leak — and ran Docent live to see the current build clear the hard slice at 3/3. Before the guided project, try extending the demos: add the auth case from Exercise 1 and re-run coverage, slice the recorded results by source_doc, and lower the duplicate threshold to watch which pairs start getting flagged. Getting a feel for how sensitive each measurement is will make Lesson 5 — where you assemble all of this into Docent’s one authoritative golden dataset — land as an obvious next step rather than a new idea.