Lesson 2 - Kinds of Checks

Welcome to Kinds of Checks#

In Lesson 1 you learned that the check matters most. This lesson is about the kinds of check you can build, because they’re not all equal. Some give you a clean, objective yes or no; others depend on judgment and need more care to stay honest. Knowing the menu — and knowing which item fits the task in front of you — is the difference between a check that reliably catches problems and one that just feels like checking.

By the end of this lesson, you will be able to:

• Name the main kinds of checks and what each is good for
• Pick the strongest check a given task allows
• Prefer objective checks where they exist, and use judgment-based ones only for the rest
• Recognize when a real-world signal is the check that actually counts

Let’s walk the menu from the most objective to the most judgment-based.

A Spectrum from Rules to Judgment#

Checks live on a spectrum. On one end are objective checks — a rule either passes or it doesn’t, and any two people (or an AI) would agree. On the other end are judgment-based checks, where “good” is a matter of assessment and reasonable people might differ. Objective checks are cheaper, faster, and harder to fool; judgment-based checks are more flexible and can handle qualities a rule can’t capture. The skill is reaching for the most objective check the task allows, and only moving toward judgment for the parts that genuinely need it.

Let’s take them one at a time.

Hard Rules, Checklists, and Reference Examples#

Hard rules (tests) are the strongest checks, because they give a clean pass or fail that can’t be argued with. “Under 250 words” is a count. “Every book named appears in the stock list” is a lookup. “The founding year says 2009” is a match. Where a rule like this exists, use it — it’s cheap, instant, reproducible, and immune to a persuasive tone. A hard rule is the closest thing you get to certainty in a loop, so build your check on as many of them as the task will support.

Checklists are a small step toward judgment: a list of concrete must-haves you confirm one by one. Most are near-objective (“leads with this week’s event,” “includes the address”), some are softer (“sounds friendly”). A checklist is the everyday workhorse of loop engineering — it’s just your definition of done from Module 2, used as a check. If you can turn a task into a checklist, you usually should.

Reference examples check by comparison: you hold the attempt up against a known-good example and ask “does this match the standard that one set?” This is powerful when quality is easier to recognize than to describe. You might not be able to fully specify a good Harborlight newsletter, but you can point at last week’s, which worked, and ask whether this week’s is as good in the ways that matter. A strong reference turns a vague standard into a concrete comparison.

Rubrics for the Soft Stuff#

Some qualities resist rules and checklists — “is the tone genuinely warm?”, “is this argument persuasive?”, “is the explanation clear?” For these you use a rubric: you name the dimensions that matter and score each one, usually on a small scale, with a short description of what each score means. Instead of one fuzzy “is it good?”, a rubric asks several smaller, more answerable questions — “warmth: 1–5,” “clarity: 1–5,” “on-brand: 1–5” — and defines what a 4 looks like versus a 2.

Rubrics are the most flexible check, and that flexibility is exactly why they need the most care. A judgment-based score can drift, get generous when you’re tired, or — when an AI does the scoring — get gamed. The next two lessons are largely about making rubric-style checks trustworthy: getting honest scores instead of flattery (Lesson 3), and separating the scorer from the doer (Lesson 4). For now, the key point is that a rubric is what you reach for after you’ve squeezed out every hard rule and checklist item you can, not instead of them.

The Real-World Signal#

There’s one more check, and in a sense it’s the only one that truly counts: did it actually work? The newsletter’s real check isn’t your rubric — it’s whether people opened it, read it, and came to the event. The product description’s real check is whether it helped someone buy. This real-world signal is the ground truth every other check is a stand-in for.

So why not just use it? Because it’s slow and it arrives too late. You can’t wait to see next week’s footfall before deciding whether to send today’s newsletter. The real-world signal can’t finish the current loop — but it’s gold for improving the next one. If last month’s “warm” newsletters got more replies than the “punchy” ones, that’s real evidence to fold back into your definition of done and your rubric. Treat the real-world signal as the feedback that calibrates your faster, in-the-moment checks over time.

Choosing for a Harborlight Task#

Take the weekly newsletter. A good check isn’t one kind — it’s a stack, using the strongest tool for each part:

• Hard rules: under 250 words; every title in the stock list; the event date matches the calendar. (Objective — build these first.)
• Checklist: leads with the main event; includes a call to action; addresses the reader as “you.” (Near-objective — quick to confirm.)
• Reference example: reads about as well as last month’s best issue. (Comparison for overall quality.)
• Rubric: warmth and on-brand tone, scored 1–5, must be at least a 4. (Judgment — only for the genuinely soft parts.)
• Real-world signal: open and reply rates, reviewed monthly, to tune all of the above.

Notice how little is left to pure judgment once the rules and checklist do their work. That’s the goal: make the objective checks carry as much of the load as possible, so the fragile, judgment-based part is small and manageable.

Practice Exercises#

Exercise 1: Match the check#

For each, name the strongest kind of check that fits: (a) “the email is under 150 words,” (b) “the apology feels sincere,” (c) “the flyer looks as polished as our last one,” (d) “the price listed matches our price sheet.”

Exercise 2: Push it left#

Someone plans to check a product description with “the AI decides if it’s good enough.” Rewrite their check to lean on rules and a checklist first, using a rubric only where needed.

Exercise 3: Name the real-world signal#

For Harborlight’s event flyer, what real-world signal would tell you it actually worked — and why can’t it finish the current loop?

Summary#

Checks live on a spectrum from objective to judgment-based, and picking well means reaching for the most objective check a task allows. Hard rules (counts, lookups, matches) give an un-foolable pass/fail and should carry as much of the check as possible. Checklists confirm concrete must-haves one by one — your definition of done, used as a check. Reference examples check by comparison to a known-good standard, useful when quality is easier to recognize than describe. Rubrics score the genuinely soft dimensions and are the most flexible but the most fragile, so they’re for the fuzzy remainder only, after rules and checklists have done their work. And the real-world signal — did it actually work? — is the ground truth every other check approximates; it’s too slow to finish the current loop but invaluable for improving the next one. A good check is usually a stack of these, using the strongest tool for each part.

Key Concepts#

• Hard rule / test — an objective pass/fail (count, lookup, match); the strongest check, use it wherever possible.
• Checklist — concrete must-haves confirmed one by one; the everyday workhorse.
• Reference example — checking by comparison to a known-good standard.
• Rubric — scored dimensions for soft qualities; flexible but fragile, for the remainder only.
• Real-world signal — the true outcome (opens, replies, sales); calibrates your faster checks over time.

Why This Matters#

Most weak checks fail because they reach for judgment when a rule was available — “the AI decides if it’s good” instead of “under 250 words and every fact matches the source.” Knowing the menu lets you build a check that’s mostly objective, which means mostly reliable, with only a small, carefully-handled judgment part. That stacking instinct is what separates a check that catches real problems from one that just feels rigorous. Next, you’ll tackle the hardest part of the judgment end: getting an AI to review work honestly, instead of telling you what you want to hear.

Continue Building Your Skills#

You now have a menu of checks and the instinct to reach for the most objective one a task allows, stacking rules and checklists before you ever lean on a rubric. That instinct alone will make your loops far more reliable. But the judgment-based checks are unavoidable for soft work — so next, you’ll learn to get honest grading out of an AI, instead of the confident flattery it hands you by default.