Lesson 4 - Failing Loudly

Welcome to Failing Loudly#

Lesson 1 gave CityFlow’s pipeline a suite of executable validation rules. Lesson 2 gave it somewhere to put the rows that fail them — a quarantine table with reasons, so nothing is ever silently dropped. Lesson 3 gave it a voice: a structured run log that can answer, tomorrow morning, exactly what last night’s run did. Every one of those pieces assumes the job keeps going. This lesson is about the case where it must not.

Because there is a decision buried in every validation suite that nobody makes explicitly until it bites them: which failures are worth stopping for? Twenty-one malformed rows in a quarter of nine and a half million is not a crisis — it is a pipeline doing its job, quarantining junk and moving on. A month that arrives at four percent of its normal size is not dirty data, it is a broken upstream export, and every number computed from it will be wrong in a way no downstream consumer can detect. The first deserves a warning. The second deserves a job that refuses to publish, exits nonzero, and makes a scheduler light up red at 2 a.m. This lesson builds the mechanism for both — severity levels and thresholds — and then proves the hard path for real: a deliberately truncated feed that fails a critical rule, aborts before the write, exits with a real nonzero code, and leaves yesterday’s verified 240,917 buckets and $256,692,373.14 sitting on disk completely untouched.

By the end of this lesson, you will be able to:

• Assign each validation rule a severity (warn or critical) and calibrate thresholds that separate ordinary dirt from a broken feed
• Explain why a job’s stages must be ordered read → validate input → transform → validate output → gate → write, and what goes wrong when the write comes first
• Return real process exit codes with sys.exit(1) so a scheduler can act on a failure, and recognize why a handler that swallows an exception and exits 0 is more dangerous than one that crashes
• Run the same gated job against a good feed and a corrupt feed and read the actual exit codes each produces
• Verify that an aborted run published nothing and left the previous good table intact

You’ll need pyspark (JDK 17 or 21 — see the course setup), the three taxi-quarter Parquet files, and Python’s standard subprocess and sys modules to launch the job the way a scheduler would.

The Calibration Problem#

Start with the observation that makes thresholds necessary. Our quarter has bad rows in it — the out-of-window strays Lesson 2 quarantined. The question a gate has to answer is not “are there bad rows?” (there always are) but “are there enough bad rows to disbelieve the whole batch?” So measure the rate.

import warnings
warnings.filterwarnings("ignore")
import os, sys, shutil, subprocess
from pathlib import Path
from pyspark.sql import SparkSession
import pyspark.sql.functions as F

spark = (SparkSession.builder
         .appName("cityflow")
         .master("local[*]")
         .config("spark.ui.showConsoleProgress", "false")
         .getOrCreate())
spark.sparkContext.setLogLevel("ERROR")

QUARTER = ["yellow_tripdata_2024-01.parquet",
           "yellow_tripdata_2024-02.parquet",
           "yellow_tripdata_2024-03.parquet"]

raw = spark.read.parquet(*QUARTER)
n_in = raw.count()

in_window = ((F.col("tpep_pickup_datetime") >= "2024-01-01") &
             (F.col("tpep_pickup_datetime") < "2024-04-01"))
n_stray = raw.filter(~in_window).count()

print(f"rows received        {n_in:>12,}")
print(f"out-of-window rows   {n_stray:>12,}")
print(f"violation rate       {n_stray / n_in:>12.6%}")
print(f"that is one bad row in every {n_in // n_stray:,}")

rows received           9,554,778
out-of-window rows             21
violation rate          0.000220%
that is one bad row in every 454,989

Twenty-one rows out of 9,554,778 — a violation rate of 0.000220%, one row in every 454,989. If your gate aborts the job on any rule that fires, this quarter never publishes at all, and the on-call engineer is paged at 2 a.m. to look at twenty-one taxi trips from 2002. Do that a few times and the team learns to ignore the alerts, which is strictly worse than having none: an alarm that cries wolf is an alarm nobody reads on the night it is right. The rate is what turns an observation into a decision, and a threshold is where you write that decision down.

Two Broken Feeds, Built for Real#

To exercise the other end of the scale we need input that is genuinely wrong, not merely dirty. The realistic upstream failure is a partial export: a transfer that died partway, or a source system that only had part of the month ready when the job fired. Nothing about the resulting file is malformed — every row in it is perfectly valid — it is just missing most of itself, which is precisely the failure that slips past row-level rules and lands in a dashboard as a real-looking collapse in ridership.

We simulate two severities of that failure by slicing March: one feed that arrived a bit short, and one that barely arrived at all.

PARTIAL, TRUNCATED = "c2m7l4_partial.parquet", "c2m7l4_truncated.parquet"
for path in (PARTIAL, TRUNCATED):
    shutil.rmtree(path, ignore_errors=True)

march = spark.read.parquet("yellow_tripdata_2024-03.parquet")
march.limit(2_556_000).write.mode("overwrite").parquet(PARTIAL)     # a short month
march.limit(120_000).write.mode("overwrite").parquet(TRUNCATED)     # a failed export

print(f"partial feed      {spark.read.parquet(PARTIAL).count():>10,} rows")
print(f"truncated feed    {spark.read.parquet(TRUNCATED).count():>10,} rows")

partial feed       2,556,000 rows
truncated feed       120,000 rows

Both files are valid Parquet full of valid trips. Only their size betrays them, which is why the volume rule from Lesson 1 is the one that catches this class of fault — and why a suite made only of row-level rules would wave both of these through.

Severity and Thresholds: Turning a Ratio Into a Verdict#

Here is the mechanism. A rule doesn’t return a boolean; it returns an observation, and a threshold maps that observation onto one of three verdicts. pass means inside tolerance. warn means outside tolerance but still plausible — record it loudly, publish anyway, let a human look in the morning. critical means the data cannot be trusted at all — publish nothing.

For a volume rule, the natural observation is the ratio of rows received to rows expected. Let’s fix two thresholds and run every feed we have through them, including the real months.

EXPECTED_PER_MONTH = 3_000_000     # a normal month of NYC yellow-taxi trips
VOLUME_WARN_AT     = 0.90          # under 90% of normal: note it, keep going
VOLUME_ABORT_AT    = 0.50          # under 50%: the export is broken, publish nothing


def verdict(ratio):
    """Map an observed share-of-normal onto a severity verdict."""
    if ratio >= VOLUME_WARN_AT:
        return "pass"
    if ratio >= VOLUME_ABORT_AT:
        return "warn"
    return "critical"


feeds = [("January, as received",   "yellow_tripdata_2024-01.parquet"),
         ("February, as received",  "yellow_tripdata_2024-02.parquet"),
         ("March, as received",     "yellow_tripdata_2024-03.parquet"),
         ("a short partial export", PARTIAL),
         ("a truncated export",     TRUNCATED)]

print("FEED".ljust(26) + "ROWS".rjust(12) + "SHARE OF NORMAL".rjust(18) + "VERDICT".rjust(11))
for label, path in feeds:
    rows = spark.read.parquet(path).count()
    ratio = rows / EXPECTED_PER_MONTH
    print(label.ljust(26) + f"{rows:>12,}" + f"{ratio:>17.1%}" + f"{verdict(ratio):>12}")

FEED                              ROWS   SHARE OF NORMAL    VERDICT
January, as received         2,964,624            98.8%        pass
February, as received        3,007,526           100.3%        pass
March, as received           3,582,628           119.4%        pass
a short partial export       2,556,000            85.2%        warn
a truncated export             120,000             4.0%    critical

The three real months land between 98.8% and 119.4% of normal and all pass — which is the first thing a threshold has to do: leave normal variation alone. March is nearly 20% above the nominal three million and that is simply what March is; a threshold tight enough to flag it would fire every quarter forever. The partial export at 85.2% is the interesting case, close enough to plausible that it could be a genuinely quiet month, so it earns a warn rather than a stop. The truncated export at 4.0% is not ambiguous at all. No real March loses 96% of its trips. That is a critical failure, and the correct behavior is to publish nothing.

Exit Codes: How a Job Tells a Scheduler It Failed#

Before wiring the gate in, get one thing straight about how a failure travels. Whatever runs this job at 2 a.m. — cron, Airflow, a shell script — is not reading your log output. It knows exactly one thing about the run: the process exit code. Zero means success. Anything else means failure, and that nonzero is what triggers the retry, the alert, and the “don’t start the downstream job” decision.

Which makes the following pattern, extremely common and entirely well-intentioned, one of the most dangerous things in a data pipeline. Both handlers below catch the same error and log it; only one of them tells the truth to the caller.

HANDLERS = '''\
import sys

def load_batch(rows):
    if rows < 2_000_000:
        raise ValueError(f"volume rule failed: {rows:,} rows")
    return rows

def swallowing_main():            # catches, logs, and carries on
    try:
        load_batch(120_000)
    except ValueError as e:
        print("ERROR:", e)        # looks responsible, tells the caller nothing
    print("run finished")
    return 0

def loud_main():                  # same log, then a nonzero exit
    try:
        load_batch(120_000)
    except ValueError as e:
        print("CRITICAL:", e)
        return 1                  # the scheduler can SEE this
    print("run finished")
    return 0

if __name__ == "__main__":
    sys.exit(swallowing_main() if sys.argv[1] == "swallow" else loud_main())
'''
Path("c2m7l4_handlers.py").write_text(HANDLERS)

for mode in ("swallow", "loud"):
    result = subprocess.run([sys.executable, "c2m7l4_handlers.py", mode],
                            capture_output=True, text=True)
    print(f"--- {mode} ---")
    print(result.stdout.rstrip())
    print("exit code:", result.returncode)

--- swallow ---
ERROR: volume rule failed: 120,000 rows
run finished
exit code: 0
--- loud ---
CRITICAL: volume rule failed: 120,000 rows
exit code: 1

Read the first one carefully, because it is the trap. The error was detected. The error was logged. A human reading the log would see it immediately. And the process exited 0 — and then, worse, printed “run finished”, because execution simply continued past the except block. To the scheduler this run is indistinguishable from a perfect night: no alert, no retry, downstream jobs released to consume whatever this one left behind. A job that swallows an exception and exits 0 is genuinely more dangerous than one that crashes, because a crash is at least honest. The loud variant logs exactly the same message and then returns 1, and that single digit is the entire difference between a failure somebody handles and a failure somebody discovers next quarter.

The rule to carry: an error you cannot correct must change the exit code. Catching an exception to add context is good practice; catching it to make the run look successful is how bad data gets published with a green checkmark next to it.

The Gated Job: Validate, Gate, Then Write#

Now the ordering, which is the real lesson. A job that writes its output and then validates it has already lost — the bad table is on disk, downstream consumers may already be reading it, and your validation step has been demoted from a gate to a postmortem. The stages must run in the order: read → validate input → transform → validate output → gate → write, with the write last and conditional on everything before it.

Here is that job as a standalone script, because a real pipeline job is a standalone script — something a scheduler invokes with arguments and judges by its exit code. It takes a comma-separated list of input feeds and an output path.

JOB_SRC = '''\
"""CityFlow zone-hour ETL with a fail-loudly gate.

Order: read -> validate input -> transform -> validate output -> GATE -> write.
The write is the LAST thing that happens, and only if no critical rule failed.
"""
import sys, warnings
warnings.filterwarnings("ignore")
from pyspark.sql import SparkSession
import pyspark.sql.functions as F

WINDOW_LO, WINDOW_HI = "2024-01-01", "2024-04-01"
EXPECTED_PER_FEED = 3_000_000     # a normal month of yellow-taxi trips
VOLUME_WARN_AT    = 0.90          # under 90% of expected: note it, keep going
VOLUME_ABORT_AT   = 0.50          # under 50%: the export is broken, publish nothing
STRAY_WARN_AT     = 0.001         # 0.1% out-of-window rows
STRAY_ABORT_AT    = 0.05          # 5% out-of-window rows

results = []   # (name, severity, passed, observed, expected)


def check(name, severity, passed, observed, expected):
    """Record one rule outcome. severity is 'warn' or 'critical'."""
    results.append((name, severity, passed, observed, expected))
    return passed


def tier(ratio, warn_at, abort_at):
    """Turn an observed ratio into a severity: below abort_at it is critical."""
    return "critical" if ratio < abort_at else "warn"


def report():
    print("RULE".ljust(24) + "SEVERITY".ljust(10) + "RESULT".ljust(8)
          + "OBSERVED".ljust(34) + "EXPECTED")
    for name, sev, passed, obs, exp in results:
        outcome = "pass" if passed else ("WARN" if sev == "warn" else "FAIL")
        print(name.ljust(24) + sev.ljust(10) + outcome.ljust(8) + obs.ljust(34) + exp)


def critical_failures():
    return [name for name, sev, passed, _, _ in results if sev == "critical" and not passed]


def main(feeds, out_path):
    spark = (SparkSession.builder
             .appName("cityflow")
             .master("local[*]")
             .config("spark.ui.showConsoleProgress", "false")
             .getOrCreate())
    spark.sparkContext.setLogLevel("ERROR")

    # ---- 1. read ---------------------------------------------------------
    raw = spark.read.parquet(*feeds)
    n_in = raw.count()

    # ---- 2. validate the INPUT -------------------------------------------
    expected = EXPECTED_PER_FEED * len(feeds)
    ratio = n_in / expected
    check("input.volume", tier(ratio, VOLUME_WARN_AT, VOLUME_ABORT_AT),
          ratio >= VOLUME_WARN_AT, f"{n_in:,} rows ({ratio:.1%} of normal)",
          f">= {VOLUME_WARN_AT:.0%} of {expected:,}")

    in_window = ((F.col("tpep_pickup_datetime") >= WINDOW_LO) &
                 (F.col("tpep_pickup_datetime") < WINDOW_HI))
    n_stray = raw.filter(~in_window).count()
    rate = n_stray / n_in if n_in else 0.0
    check("input.window", "critical" if rate >= STRAY_ABORT_AT else "warn",
          rate < STRAY_WARN_AT, f"{n_stray:,} stray ({rate:.6%})",
          f"< {STRAY_WARN_AT:.1%} of rows")

    # ---- 3. transform -----------------------------------------------------
    kept = raw.filter(in_window)
    zone_hour = (kept
        .withColumn("pickup_hour", F.date_trunc("hour", "tpep_pickup_datetime"))
        .groupBy("PULocationID", "pickup_hour")
        .agg(F.count("*").alias("trips"), F.sum("total_amount").alias("revenue"))
        .withColumnRenamed("PULocationID", "zone")).cache()

    agg = zone_hour.agg(F.count("*").alias("buckets"),
                        F.sum("trips").alias("trips"),
                        F.sum("revenue").alias("revenue")).first()
    n_buckets, n_kept, revenue = agg["buckets"], agg["trips"], agg["revenue"]
    n_quarantined = n_in - n_kept

    # ---- 4. validate the OUTPUT -------------------------------------------
    check("output.reconciles", "critical", n_kept + n_quarantined == n_in,
          f"{n_kept:,} + {n_quarantined:,}", f"== {n_in:,}")
    check("output.nonempty", "critical", n_buckets > 0,
          f"{n_buckets:,} buckets", "> 0")

    report()
    print("")
    print(f"summary  in={n_in:,}  kept={n_kept:,}  quarantined={n_quarantined:,}  "
          f"buckets={n_buckets:,}  revenue=${revenue:,.2f}")

    # ---- 5. THE GATE -------------------------------------------------------
    failed = critical_failures()
    if failed:
        print("ABORT: critical rule(s) failed: " + ", ".join(failed))
        print("nothing written; the published table is left exactly as it was")
        spark.stop()
        return 1

    warned = [n for n, s, p, _, _ in results if s == "warn" and not p]
    if warned:
        print("WARN: " + ", ".join(warned) + " outside tolerance; publishing anyway")

    # ---- 6. write (reached only when the gate passes) ----------------------
    zone_hour.write.mode("overwrite").parquet(out_path)
    print("WROTE " + out_path)
    spark.stop()
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1].split(","), sys.argv[2]))
'''
Path("c2m7l4_job.py").write_text(JOB_SRC)
print("wrote c2m7l4_job.py:", len(JOB_SRC.splitlines()), "lines")

wrote c2m7l4_job.py: 116 lines

Three details in that script are worth naming. First, check() records a result rather than raising, so the suite runs to completion and the report shows every rule’s status — you want to know about all four problems, not just the first one. The abort still happens before the write, which is the property that matters. Second, critical_failures() filters on severity, so a failing warn rule can never stop the job and a failing critical rule always does; the severity is attached at the point the rule is evaluated, where the context to judge it lives. Third, main() returns an integer that sys.exit() hands to the operating system — no bare exit(), no raise swallowed by an outer handler, just a value that becomes the process’s exit code.

Run One: The Good Feed#

Run the job the way a scheduler would: as a subprocess, with arguments, and inspect what comes back.

OUT = "c2m7l4_report"


def run_job(label, feeds, out_path):
    result = subprocess.run([sys.executable, "c2m7l4_job.py", ",".join(feeds), out_path],
                            capture_output=True, text=True)
    print(f"===== {label} =====")
    print(result.stdout.rstrip())
    print("EXIT CODE:", result.returncode)
    return result.returncode


good_code = run_job("GOOD RUN: the full quarter", QUARTER, OUT)

===== GOOD RUN: the full quarter =====
RULE                    SEVERITY  RESULT  OBSERVED                          EXPECTED
input.volume            warn      pass    9,554,778 rows (106.2% of normal) >= 90% of 9,000,000
input.window            warn      pass    21 stray (0.000220%)              < 0.1% of rows
output.reconciles       critical  pass    9,554,757 + 21                    == 9,554,778
output.nonempty         critical  pass    240,917 buckets                   > 0

summary  in=9,554,778  kept=9,554,757  quarantined=21  buckets=240,917  revenue=$256,692,373.14
WROTE c2m7l4_report
EXIT CODE: 0

Every rule passes, the table is written, and the process returns exit code 0. The numbers cross-verify against Course 1’s ground truth exactly — 240,917 buckets, 9,554,757 trips kept, 21 quarantined, $256,692,373.14 — which is the whole point of the output validation stage: the gate is not checking that the job ran, it is checking that the job produced the shape of answer this pipeline is supposed to produce. Note also that the 21 strays passed the window rule outright rather than warning: at 0.000220% they are nearly three orders of magnitude below the 0.1% warn threshold, exactly the calibration we argued for above.

Run Two: A Warning That Doesn’t Stop Anything#

Now the short feed — 85.2% of a normal month. This is the case that justifies having two severities at all.

warn_code = run_job("DEGRADED RUN: a short partial export",
                    [PARTIAL], "c2m7l4_partial_report")

===== DEGRADED RUN: a short partial export =====
RULE                    SEVERITY  RESULT  OBSERVED                          EXPECTED
input.volume            warn      WARN    2,556,000 rows (85.2% of normal)  >= 90% of 3,000,000
input.window            warn      pass    2 stray (0.000078%)               < 0.1% of rows
output.reconciles       critical  pass    2,555,998 + 2                     == 2,556,000
output.nonempty         critical  pass    62,661 buckets                    > 0

summary  in=2,556,000  kept=2,555,998  quarantined=2  buckets=62,661  revenue=$70,497,153.50
WARN: input.volume outside tolerance; publishing anyway
WROTE c2m7l4_partial_report
EXIT CODE: 0

The volume rule failed — WARN in the results table, and an explicit WARN: line before the write — and the job published anyway and exited 0. That is the designed behavior, not a bug: 85.2% of normal is suspicious, not impossible, and a pipeline that halts on every suspicion is a pipeline that halts constantly. The failure is recorded where tomorrow’s reader will find it (this is exactly what Lesson 3’s structured log is for), the data is available in the meantime, and a human decides whether the shortfall was a quiet week or a transfer that needs re-running. Note the slice carries 2 out-of-window strays of its own, and the reconciliation still balances: 2,555,998 + 2 = 2,556,000.

Run Three: The Abort, and What Survives It#

Now the real test. Feed the job the truncated export — 120,000 rows, 4.0% of a month — and point it at the same output path the good run just published to. If the gate works, that table must come through untouched.

bad_code = run_job("BAD RUN: a truncated export", [TRUNCATED], OUT)
print("\ngood run exit code:", good_code,
      "| degraded:", warn_code, "| bad:", bad_code)

===== BAD RUN: a truncated export =====
RULE                    SEVERITY  RESULT  OBSERVED                          EXPECTED
input.volume            critical  FAIL    120,000 rows (4.0% of normal)     >= 90% of 3,000,000
input.window            warn      pass    0 stray (0.000000%)               < 0.1% of rows
output.reconciles       critical  pass    120,000 + 0                       == 120,000
output.nonempty         critical  pass    2,891 buckets                     > 0

summary  in=120,000  kept=120,000  quarantined=0  buckets=2,891  revenue=$3,191,378.53
ABORT: critical rule(s) failed: input.volume
nothing written; the published table is left exactly as it was
EXIT CODE: 1

good run exit code: 0 | degraded: 0 | bad: 1

Look at what the three other rules say. The window rule passed — zero strays. The reconciliation passed — 120,000 in, 120,000 out, perfectly balanced. The output is non-empty — 2,891 real buckets and $3,191,378.53 of real revenue, every dollar of it genuine. A batch can be internally consistent and completely wrong, and that is exactly why the volume rule exists: it is the only rule here comparing the batch against what a month is supposed to look like, rather than against itself. Without it, this run publishes 2,891 buckets over the top of 240,917 and every March dashboard drops 96% overnight with nothing in the logs to explain it.

Instead the gate caught it, the job aborted before reaching step 6, and the process returned exit code 1. Now confirm the part that matters — that “aborted before the write” means what it says:

published = spark.read.parquet(OUT)
row = published.agg(F.count("*").alias("buckets"),
                    F.sum("trips").alias("trips"),
                    F.sum("revenue").alias("revenue")).first()
print("the published table, read back AFTER the aborted run:")
print(f"  buckets   {row['buckets']:>12,}")
print(f"  trips     {row['trips']:>12,}")
print(f"  revenue  ${row['revenue']:>12,.2f}")
print("  matches Course 1's ground truth:",
      row["buckets"] == 240_917 and row["trips"] == 9_554_757
      and round(row["revenue"], 2) == 256_692_373.14)

the published table, read back AFTER the aborted run:
  buckets        240,917
  trips        9,554,757
  revenue  $256,692,373.14
  matches Course 1's ground truth: True

Untouched. The failed run pointed at this exact path and left it at 240,917 buckets, 9,554,757 trips, $256,692,373.14 — still Course 1’s ground truth to the cent. Downstream consumers reading this table during the failed run saw yesterday’s correct answer, not a partial or corrupted one. That is the entire payoff of putting the gate before the write: the worst outcome of a bad night is stale data plus a red alert, and stale-with-an-alert beats wrong-and-silent every single time.

spark.stop()

for path in [PARTIAL, TRUNCATED, OUT, "c2m7l4_partial_report",
             "c2m7l4_job.py", "c2m7l4_handlers.py", "spark-warehouse"]:
    if os.path.isdir(path):
        shutil.rmtree(path, ignore_errors=True)
    elif os.path.isfile(path):
        os.remove(path)

leftovers = [p for p in os.listdir(".") if p.startswith("c2m7l4_")]
print("cleaned:", leftovers or "all temp files removed")
print("session closed")

cleaned: all temp files removed
session closed

Practice Exercises#

Exercise 1 — Move the gate and watch the damage. Copy c2m7l4_job.py to a second script and deliberately break the ordering: move the zone_hour.write(...) call to just after the transform, before the output validation and before the gate, leaving the gate’s return 1 where it is. Run the good feed to publish a clean table, then run the truncated feed against the same output path. Confirm that the job still exits 1 — and that the published table is now the bad run’s 2,891 buckets rather than 240,917. Write one sentence explaining what the exit code is worth in that version.

Exercise 2 — Calibrate a threshold against real variation. The lesson fixed EXPECTED_PER_MONTH at a round 3,000,000, but the real months are 2,964,624 / 3,007,526 / 3,582,628. Compute their mean and the share-of-mean each month represents, then find the tightest VOLUME_WARN_AT that still lets all three pass. Compare it to the 0.90 the lesson used, and decide which you would ship for a table that drives a public dashboard versus one that drives payouts.

Exercise 3 — Add a critical rule that catches a plausible-looking corruption. Build a third bad feed from March that is full size but has total_amount multiplied by 100 for every row. It will sail past the volume rule and past the window rule. Add an output.revenue_per_trip rule to the job that computes mean revenue per trip and fails as critical when it falls outside a sane band (the real quarter runs about $26.87 per trip — derive it, don’t take the number on faith), then run the corrupted feed and confirm the job aborts with exit code 1 and publishes nothing.

Summary#

A validation suite is only half a safety system; the other half is deciding what to do when a rule fails, and the answer is not always “stop”. We calibrated that judgment on the real quarter: 21 out-of-window rows in 9,554,778 is a violation rate of 0.000220%, one row in 454,989, which quarantine absorbs without waking anyone — while a feed arriving at 4.0% of a normal month is a broken upstream export whose every number is wrong. The mechanism that separates them is severity plus thresholds: a volume rule with a warn line at 90% of normal and an abort line at 50% left all three real months alone (98.8%, 100.3%, 119.4% of nominal), flagged a 2,556,000-row partial export at 85.2% as warn, and condemned a 120,000-row truncated export at 4.0% as critical. We then wired a gate into the job in the only order that protects anything — read → validate input → transform → validate output → gate → write — and ran it three times as a real subprocess. The good quarter passed everything, wrote 240,917 buckets and $256,692,373.14, and returned exit code 0. The degraded feed failed the volume rule at warn, logged it, published its 62,661 buckets anyway, and still returned exit code 0. The truncated feed failed it as critical, aborted before step 6, and returned exit code 1 — and reading the target table afterward still gave 240,917 / 9,554,757 / $256,692,373.14, untouched. The bad run was internally consistent throughout (120,000 rows in, 120,000 reconciled, 2,891 non-empty buckets, $3,191,378.53 of genuine revenue), which is precisely why a rule comparing the batch to expected volume was the only thing standing between it and a published table. Finally, exit codes are how any of this reaches a scheduler: a handler that logs an error and returns 0 printed “run finished” and looked like a perfect night, while the identical handler returning 1 raised a real failure.

Key Concepts#

• Severity — every rule carries a warn or critical label assigned where the rule is evaluated. A failing warn rule records the problem and lets the job publish; a failing critical rule stops it. Without the distinction you get either a job that never runs or a job that never stops.
• Thresholds — the numbers that turn an observation into a verdict, and a product decision rather than a technical one. They must leave normal variation alone (March at 119.4% of nominal is just March) while catching real breakage (4.0% is not), and they should be set with whoever owns the cost of a wrong answer.
• Abort before the write — the stage ordering read → validate input → transform → validate output → gate → write is what makes a gate a gate. Validate after writing and you have a postmortem, not a guard: the aborted run left the published table at 240,917 buckets exactly because nothing after step 5 executed.
• Exit codes — a scheduler judges a run by its process exit code and nothing else. sys.exit(1) on critical failure is what triggers the alert, the retry, and the hold on downstream jobs; a handler that catches an exception, logs it, and exits 0 is more dangerous than a crash, because it reports success while publishing nothing usable.
• Internally consistent but wrong — the truncated feed passed the window rule, the reconciliation, and the non-empty check. Rules that only compare a batch to itself cannot detect a batch that is entirely missing; catching that requires comparing against an external expectation of volume, ratio, or total.

Why This Matters#

The failures that hurt most in data engineering are rarely the ones that crash. A crash gets noticed within the hour. The expensive failure is the run that completed successfully, exited 0, published a table that is 96% short or 100x inflated, and let three days of decisions be made on it before anybody asks why the numbers look strange — by which point the pipeline has run four more times, the bad data is upstream of six other tables, and reconstructing what happened takes longer than the original outage would have. Everything in this lesson exists to convert that silent, expensive failure into a loud, cheap one: a gate before the write so nothing wrong is published, thresholds calibrated so alerts stay believable, and a nonzero exit code so the machinery that runs your job at 2 a.m. actually knows. It is a small amount of code for a large change in what “the pipeline ran last night” is worth as a statement.

Continue Building Your Skills#

Module 7’s four pieces are now complete: rules that state what valid means (Lesson 1), quarantine that keeps the bad rows instead of dropping them (Lesson 2), a structured run log that explains what happened (Lesson 3), and a gate with real exit codes that decides whether to publish at all (this lesson). Lesson 5, the guided project A Guarded ETL Run, is where they stop being four separate demos and become one job you could genuinely schedule: Module 6’s quarterly ETL with the full guard rail assembled around it — input validation, quarantine written alongside the good data, the transform, output validation that reconciles and cross-verifies, the gate, and a structured run summary emitted either way. You’ll prove both paths end to end one more time, on the numbers you now know by heart: a clean run reaching 240,917 buckets, 9,554,757 trips, $256,692,373.14 and exiting 0, and an injected fault aborting before the write with the failing rule named in the log and a nonzero code on the way out. Bring the ordering from this lesson with you, because it is the backbone of that script — and bring the exit code too, since Module 8 is where something on the other end finally acts on it.