What you'll build

You'll rejoin CityFlow one last time and give their pipeline the piece it's been missing: a single place that runs every step in the right order, on a schedule, with retries, visibility, and alerts when something breaks. Starting from why hand-rolled scripts and cron eventually fall over, you'll install a real Apache Airflow stack (Docker, LocalExecutor), write real DAGs with the TaskFlow API and classic operators, wait on and talk to real systems with sensors and hooks, and rebuild CityFlow's ingest→validate→transform→load pipeline as a tested, logged, monitored DAG. The capstone ties this whole four-course path together: a real Airflow DAG submits a real PySpark job (Course 2) to the real Kubernetes cluster (Course 3), end to end, entirely on your own machine.

Before you start

You'll need comfortable Python and a working command line. This course assumes you've either taken Docker & Kubernetes for Data Engineering and PySpark for Data Engineering already, or are comfortable picking up Docker and a running Kind cluster as you go — the capstone reuses both directly. No prior Airflow experience is assumed; Module 2 starts from installing it.

Set up your environment

Everything in this course runs for real, locally, on your own machine — no cloud account, nothing to pay for. You'll need a computer with a few gigabytes of free disk space and an internet connection.

1. Docker Desktop — if you haven't already installed it in the Docker & Kubernetes course, Module 2 walks you through running Airflow with it.
2. A local Kind cluster — reused from the Docker & Kubernetes course for the capstone; instructions to recreate it are included if you don't already have one.

Every command in every lesson is real and was actually run to produce the output you'll see — nothing is simulated or invented.