Welcome to Running a Data Job on Kubernetes, the seventh module of Docker & Kubernetes for Data Engineering. Module 5 gave you the vocabulary - Pods, Deployments, Services - and Module 6 gave CityFlow’s workloads config, storage, and a real run-to-completion shape. This module takes CityFlow’s actual ETL job, the one Module 3 first containerized and Module 4 ran through a real, healthcheck-gated Compose stack, and moves it onto Kubernetes for real, end to end.

You’ll start by translating that exact Compose stack object by object - a Postgres service becomes a Deployment, a Service, and a PersistentVolumeClaim; the ETL service becomes a Job - and hit the one genuine gap Kubernetes leaves open: there is no native equivalent of Compose’s depends_on: condition: service_healthy, so you’ll build the honest replacement yourself with an initContainer. From there you’ll give the scheduler the two numbers it actually needs, and measure, rather than guess, what a container does when it crosses each one: a memory limit kills, real Exit Code 137, OOMKilled; a CPU limit only throttles, a real measured 28x slowdown with the identical correct answer. You’ll diagnose two completely real failures - a genuine KeyError crash and a genuine unschedulable pod - with the three-tool sequence kubectl logs, describe, and events, learning exactly which of those tools has anything to show you when a container never even starts.

The module closes with CityFlow’s ETL job in its final, production-shaped form: a real image, its dataset URL and time window externalized into a ConfigMap, resource requests and limits sized from this module’s own measurement, and a guided project that runs the whole pipeline - PVC, Deployment, Service, ConfigMap, Job - from a clean cluster state, verified three independent ways against the same real January 2024 numbers this course has checked since Module 3: 77,530 zone-hour buckets, 2,964,606 trips, $79,455,941.50 in total revenue.

Start with Lesson 1, and see exactly how much of a real Compose stack translates to Kubernetes mechanically - and exactly where it doesn’t.