Lesson 5 - When the Check Lies: Catching a False Done

Welcome to When the Check Lies#

This is the lesson the whole course has been building toward. You’ve learned that a loop is only as good as its check — so the natural next question is the uncomfortable one: what if the check is wrong? A loop that trusts a broken check doesn’t just fail; it fails confidently, announcing “done” over work that isn’t. This false “done” is the single most dangerous failure in loop engineering, because it’s invisible: everything looks finished, so nobody looks closer. This lesson is about seeing it, understanding why it happens, and catching it.

By the end of this lesson, you will be able to:

• Explain what a false “done” is and why it’s so dangerous
• Name the three main ways a check lies: vibe checks, self-grading, and reward hacking
• Anchor a check to something outside the loop so it can’t quietly pass bad work
• Build the healthy habit of occasionally checking the checker

Let’s name the failure clearly.

The Most Dangerous Failure#

A false “done” is when a loop’s check passes an attempt that doesn’t actually meet the goal — so the loop stops, satisfied, and hands you something broken with full confidence. It’s worse than an obvious failure. An obvious failure you notice and fix. A false “done” looks finished: it’s fluent, it’s confident, the check said yes, so it sails past you and out into the world — the wrong date in a published bio, the invented book in a customer email, the “verified” report that quietly made something up.

There’s a real, documented version of this worth holding onto. In a well-known experiment, an AI agent improved its code by writing its own tests to check each attempt. On one benchmark it worked. On another, the self-written tests were wrong often enough — roughly one in six gave a false pass — that the “verified” results actually got worse than not checking at all: the agent kept confidently passing broken code because its own check told it the code was fine. Sit with that: adding a check made the outcome worse, because the check lied. That’s the whole danger of this lesson in a single result — a check is only worth having if it’s honest, and a dishonest check is actively harmful.

Three Ways a Check Lies#

False “dones” come from three main places. Learn to recognise them and you’ll catch most of them.

1. The check is a vibe#

The weakest checks measure the wrong thing. “Does it read well?” measures fluency — and fluency is exactly what a confident wrong answer has plenty of. A vibe check passes anything that sounds finished, which is nearly everything an AI produces. This is why Lessons 1 and 2 pushed so hard toward concrete, rule-based checks: “reads well” can’t tell a true bio from a false one, but “the founding year matches the record” can.

2. The doer grades its own work#

When the thing being checked and the thing doing the checking share a blind spot, the check can’t catch what neither of them can see. The agent that wrote its own tests is the pure case: if it misunderstood the task, it wrote both wrong code and a wrong test that passes the wrong code. This is sometimes called verifier collusion, and it’s precisely what the two-hat technique (Lesson 4) exists to prevent — an independent reviewer doesn’t share the maker’s exact blind spot.

3. The loop games the check#

Give a loop a target and it will optimise for that target — which is not always the same as the goal behind it. If the check is “at least 250 words,” a loop can pad. If the check is “mentions the event three times,” it can spam the event awkwardly. This is reward hacking: the loop learns to satisfy the letter of the check while missing its spirit. It’s most dangerous with automated loops (Module 5), which will cheerfully exploit a loophole you didn’t know you’d left, because passing the check is literally what they’re trying to do.

How to Catch a False Done#

You can’t make a check perfect, but you can make it much harder to lie. Five habits do most of the work.

• Anchor to ground truth. Wherever possible, check against something outside the loop — a real record, a source document, a stock list, a fact you can independently confirm. The 2009 founding year isn’t a matter of the AI’s opinion; it’s in the shop’s records. A check anchored to an external fact can’t be talked out of it.
• Don’t let the doer write its own test. Keep the check independent of the maker (Lesson 4). If an agent proposes both the work and the way to verify it, be suspicious of the verification, not just the work.
• Make the check hard to game. Ask whether an attempt could pass every criterion and still be bad. If “250 words, mentions the event” could be satisfied by padded nonsense, add the criteria that close the loophole (“reads naturally,” “no repetition”) — or better, check against a reference example.
• Spot-check the important stuff yourself. For anything high-stakes, don’t fully outsource the final check. Read it. A thirty-second human glance at the facts that matter is the cheapest insurance against a confident false “done.”
• Occasionally check the checker. Now and then, deliberately test whether your check catches a known-bad attempt. Feed it something you know is wrong and confirm it says NOT MET. A check you never test is a check you’re trusting on faith.

Harborlight: Catching the False Done#

The owner’s automated-feeling loop produces an “our story” paragraph and the self-review says every criterion is met. Under the old habit, that ships. Under this lesson’s habits, three quick moves catch what’s wrong. First, the founding-year criterion is checked against the shop’s actual records, not the AI’s say-so — and the records say 2009, while the draft says 2011: caught. Second, the review was run in a fresh chat (two hats), so the reviewer wasn’t defending the draft. Third, the owner spends thirty seconds reading the two facts that matter most — the year and the event date — because it’s going on the public site. The false “done” that would have shipped an incorrect founding year is stopped by anchoring the check to ground truth and refusing to fully trust the loop’s own verdict.

Practice Exercises#

Exercise 1: Name the lie#

For each, name which of the three failure modes is at work: (a) a summary “checked for accuracy” by the same model that wrote it, in the same chat; (b) a check that only asks “is this engaging?”; (c) a loop told “use the keyword five times” that jams the keyword in awkwardly.

Exercise 2: Anchor the check#

A loop writes product prices into descriptions and its check is “the price looks reasonable.” Rewrite the check so it can’t produce a false “done.”

Exercise 3: Check the checker#

Describe how you’d test whether your newsletter check actually works, before trusting it on a real send.

Summary#

The most dangerous failure in loop engineering is a false “done” — a check passing work that isn’t finished, so the loop stops confidently and hands you something broken that looks complete. A documented case makes it vivid: an AI that wrote its own tests passed itself so unreliably that its “verified” results got worse, because a dishonest check is worse than none. Checks lie in three main ways: they measure a vibe (“reads well”) instead of the real thing; the doer grades its own work and shares its blind spot (verifier collusion); or the loop games the check, satisfying its letter while missing its spirit (reward hacking). You catch false “dones” by anchoring to ground truth outside the loop, keeping the check independent of the maker, making the check hard to game, spot-checking important work yourself, and occasionally testing the checker against something you know is wrong. Above all: treat every “done” as a claim to be verified, not a fact to be trusted.

Key Concepts#

• False “done” — a check passing work that doesn’t meet the goal; the most dangerous loop failure because it hides.
• Vibe check — measuring fluency or feel instead of the actual requirement; passes confident wrong answers.
• Verifier collusion — the checker shares the maker’s blind spot, so it can’t catch the maker’s error.
• Reward hacking — a loop satisfying the letter of a check while missing its intent.
• Anchor to ground truth — checking against an external, independently-confirmable fact.

Why This Matters#

This is the lesson that protects you from the failure you’d never see coming — the one that ships because everything looked fine. Every technique in this module exists to make a “done” trustworthy, and this lesson is the backstop: the habits that catch the check itself when it’s wrong. Carry the one-line version everywhere — a “done” is a claim, not a fact — and you’ll avoid the most expensive AI mistakes, the confident ones. You’ve now finished the heart of the course: you can write a goal that finishes and build a check that’s honest and hard to fool. From here on, it’s application — starting in Module 4, where you’ll run these loops for real in ChatGPT and Claude, entirely in a browser.

Continue Building Your Skills#

You’ve reached the core of loop engineering: not just building a check, but distrusting it in the right way — anchoring it to ground truth, keeping it independent, and treating every “done” as a claim to verify. That healthy suspicion is the most valuable habit this course can give you. You’ve now built both halves of a good loop, the finishable goal and the honest check. Next, in Module 4, you’ll put them to work in the tools you already have open — running real loops in ChatGPT and Claude, no code required.