Welcome to Orchestrating ETL, Module 5 of Orchestrating Data Pipelines with Airflow. Every DAG so far has been a learning exercise — small, deliberately scoped to one idea at a time. This module is where that changes: you’ll rebuild CityFlow’s real pipeline as a DAG shaped the way a production ETL job actually looks, and run it against real data at a scale this course hasn’t used before.

The shape is ingest → validate → transform → load, and Lesson 1 doesn’t just name that shape — it shows why the boundaries sit exactly there, with a real DAG run that fails on purpose partway through to prove a validation failure stops the pipeline before any transform work is wasted. Lessons 2 through 4 build each stage for real: ingest reads a genuine slice of CityFlow’s actual January 2024 taxi data straight off a ~50MB parquet file, validate runs five genuine data-quality checks and writes its findings to a real JSON report and a real quarantine CSV instead of just printing a count, transform joins and aggregates the validated rows into a real per-day, per-borough summary, and load writes that summary to Postgres through the same PostgresHook and cityflow_postgres Connection Module 4 set up, with a real SELECT COUNT(*) proving it landed.

The guided project scales the exact same four-task DAG from Lessons 2-4’s one real day up to three real days — roughly 239,000 real trips, CityFlow’s largest single dataset yet — triggers it end to end, and verifies the finished run two independent ways: a real Grid and Graph screenshot, and a real Postgres query whose totals tie all the way back to validate’s own reported pass count.

Every task in this module’s DAGs is real Python running against real data on the same live Airflow 2.10.4 stack Modules 2-4 used — nothing here is a diagram of what an ETL pipeline would do.

Start with Lesson 1, and watch a real validation failure stop a real DAG before it wastes any work.