Docker & Kubernetes for Data Engineering

What you'll build

You'll rejoin CityFlow, the same mobility-analytics team from the Data Engineering path, and make sure the pipeline that produces their real, public New York City yellow-taxi reports runs identically no matter whose machine, server, or cluster it lands on. Starting from a real Docker Desktop install, you'll learn what an image and a container actually are, write your own Dockerfiles, slim and multi-stage them for real measured size wins, run multi-service stacks with Docker Compose, and orchestrate them for real on a local Kubernetes cluster (via Kind — Kubernetes in Docker) — deployments, services, config, storage, and scheduled batch jobs. The capstone containerizes CityFlow's real ETL job and runs it as a scheduled Kubernetes CronJob, entirely on your own laptop.

Before you start

You'll need comfortable Python (reading a script, running it, basic dependencies) and a working command line. If you took Scaling Python for Data Engineering or PySpark for Data Engineering, you'll recognize CityFlow's data and pipeline immediately — but neither is required to start here. No prior Docker or Kubernetes experience is assumed; Lesson 1 starts from installing Docker Desktop itself.

Set up your environment

Everything in this course runs for real, locally, on your own machine — no cloud account, nothing to pay for. You'll need a computer with a few gigabytes of free disk space and an internet connection.

1. Docker Desktop — Lesson 1 walks you through installing it on macOS, Windows, or Linux.
2. Kind (Kubernetes in Docker) — introduced when the course reaches Kubernetes, installed with a single command.

Every command in every lesson is real and was actually run to produce the output you'll see — nothing is simulated or invented.