Welcome to the Capstone, the eighth and final module of Orchestrating Data Pipelines with Airflow – and the last module of the entire four-course Data Engineering path.

Every module in this course added one layer: what a pipeline is and why cron stops being enough, Airflow’s own architecture, authoring real DAGs with the TaskFlow API, waiting on and talking to real systems with sensors and hooks, rebuilding CityFlow’s ingest-validate-transform-load pipeline as an orchestrated DAG, making that DAG trustworthy with tests and monitoring and alerts, and finally handing real work to a real Kubernetes cluster and watching it run. This module assembles every one of those pieces into a single, real, end-to-end pipeline: an Airflow DAG that submits a real PySpark job to the real datatweets-docker-k8s Kind cluster, processes the full first quarter of 2024 – 9,554,778 real trips across three real files – and verifies its own answer against a number this path has already trusted for three courses running.

That number is the whole point. Scaling Python for Data Engineering verified it with pandas and multiprocessing. PySpark for Data Engineering verified it again with a distributed Spark engine on a laptop. Docker & Kubernetes for Data Engineering verified it a third time with a real, repeatedly-firing Kubernetes CronJob. This capstone verifies it a fourth independent way – a real Airflow task submitting a real Kubernetes Job running a real PySpark computation, with a final task that doesn’t just print the answer but asserts it: 240,917 zone-hour buckets, 9,554,757 trips, $256,692,373.14 in revenue, busiest single zone-hour zone 79 at 2024-02-25T01 with 846 trips.

Start the lesson, and watch the whole Data Engineering path close its own loop.