Welcome to the Capstone, the eighth and final module of Docker & Kubernetes for Data Engineering — one project, and the argument the whole course has been building toward.

Every module since Module 1 has added one layer: a real image, a slimmed and multi-staged build, a Compose stack with a genuine healthcheck, the vocabulary of Pods and Deployments and Services on a real Kind cluster, ConfigMaps and Secrets and PersistentVolumeClaims, real Jobs and CronJobs, and finally CityFlow’s actual ETL packaged with measured resource limits and run end to end as a one-off Kubernetes Job. This module takes every one of those pieces and assembles them into a single, scheduled, self-running pipeline: CityFlow’s zone-hour ETL, scaled up from one month to the full first quarter of 2024 — 9,554,778 real trips across three real files — containerized, configured entirely through a ConfigMap, caching its own input on a PersistentVolumeClaim so a repeatedly-firing CronJob never re-downloads 160 MB it already has, sized with resource limits from a real measured memory boundary, and scheduled to run on its own with no command from you after the first apply.

The proof this module insists on is the same one every course in this path has insisted on: the report this CronJob produces, read back independently after its pods are long gone, has to match 240,917 zone-hour buckets, 9,554,757 trips, and $256,692,373.14 in revenue — the exact numbers Scaling Python for Data Engineering verified with pandas and multiprocessing, and PySpark for Data Engineering verified again with a completely different distributed engine. This module verifies them a third time, with a third completely different mechanism: a real, repeatedly-firing Kubernetes CronJob.

Start the lesson, and watch CityFlow’s whole pipeline cross its last layer.