Welcome to Testing, Logging & Monitoring DAGs, Module 6 of Orchestrating Data Pipelines with Airflow. cityflow_etl_dag.py works — Module 5 proved that with a real ~239,000-row run, a real Postgres load, and a real screenshot of four green boxes. Working isn’t the same as trustworthy. Nobody watches this DAG live at 3 a.m., so the pipeline itself has to be able to prove its own logic is correct, tell you exactly what happened when it breaks, show you where the time went, and say something the instant it fails. That’s this module.

Lesson 1 pulls validate_trips’s five real data-quality checks out into a plain, importable function and unit-tests it in total isolation — no scheduler, no worker, not even Airflow installed — and measures the real difference: a real test suite finishing in 200 milliseconds against a real DAG-trigger-and-poll cycle that took 6.68 real seconds for a three-task DAG doing almost nothing. Lesson 2 goes past the happy-path log tour Module 2 already gave you: a task that genuinely fails, and its real traceback read straight out of the log file Airflow’s own task process wrote. Lesson 3 triggers a real four-task DAG over CityFlow’s entire real January 2024 month — 2,964,624 real rows — specifically so the Grid page’s Task Duration and Gantt views have a real spread of durations worth reading. Lesson 4 wires a real on_failure_callback to a real failure and proves the alert record it writes captures that failure’s real identity, not a placeholder.

The guided project brings all three together: cityflow_etl_instrumented_dag.py, a real variant of Module 5’s own ETL DAG, instrumented with Lesson 1’s tested validation logic and Lesson 4’s real callback, triggered twice — once against a genuinely bad path to prove the alert fires exactly where it should, once against the real data to prove it stays silent when nothing is wrong.

Start with Lesson 1, and watch a real 33x gap between testing logic in isolation and testing it by running the whole DAG.