Data Quality & Logging

Welcome to Data Quality & Logging, the seventh module — the difference between a job that runs and a job you can trust. Module 6 built CityFlow’s quarterly ETL: an explicit schema, a clean read-clean-transform body, a partitioned write, and idempotent re-runs. Run it by hand and watch the numbers, and it is correct. But that job’s real life is at 2 a.m., unattended, against whatever the upstream feed dropped that night — and the feed will eventually drop something wrong. A month arrives with a tenth of its usual rows because an export failed halfway. A column’s units change. A batch of records lands with timestamps from 2002. The pipeline, as built, will process all of it perfectly and publish numbers that are perfectly wrong.

Three habits separate a pipeline that merely runs from one you can leave alone, and each is a lesson here. The first is validation: rules written as executable assertions about what the data must be — roughly this many rows, distances in a plausible range, every zone id resolvable against the dimension, these columns never null — evaluated every run and reported as a pass/fail table. The second is quarantine: when rows do violate a rule, they are not silently filtered away. They are written to their own table with the reason they failed, so good plus quarantined always reconciles back to the input, and so someone can look at what was rejected and decide whether the rule or the data was wrong. The third is logging: a structured run record — per-stage counts in and out, durations, a machine-readable summary — that lets you answer “what did last night’s run actually do?” without re-running anything.

Then the judgment the whole module builds toward: when to warn and when to abort. A handful of bad rows is a quarantine and a warning; a third of the batch failing means the upstream feed is broken and publishing anything would be worse than publishing nothing. The job needs a gate that stops before the write and exits nonzero, because a scheduler only knows a run failed if the run says so — and a job that swallows its errors and exits 0 is more dangerous than one that crashes. The guided project wires all four into the quarterly ETL and proves both paths for real: a clean run that publishes and exits 0, and a deliberately corrupted run that aborts before writing a single file, leaving the last good table untouched. Start with Lesson 1, where a rule that never fires turns out to be as suspicious as one that always does.