Title here
Summary here
Module · 5 lessons
Docker Compose
One declarative file replaces hand-typed docker run commands: persistent volumes, automatic networking, and a real multi-service data stack.
At a glance
Level
Intermediate
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Docker Compose, the fourth module of Docker & Kubernetes for Data Engineering. Modules 1 through 3 ran every container by hand — a network you created yourself, matching --network and -e flags typed across several separate docker run commands, every one of them having to agree with the others. This module replaces that with one declarative file and two commands: docker compose up and docker compose down.
You’ll start by rebuilding the exact multi-container setup from Module 1’s guided project both ways — by hand again, then as a single docker-compose.yml — so the contrast is felt, not just described, and you’ll hit a real depends_on timing gotcha along the way. From there you’ll learn where a container’s data actually lives, and watch a Postgres container lose everything on a recreate without a named volume, then keep everything with one. You’ll inspect the network Compose creates automatically, with no docker network create anywhere, and prove two containers resolve each other by service name. Then you’ll bring your own built image into a stack for the first time — a real Python job, aggregating 200,000 real CityFlow trips, running alongside Postgres in one file.
The module closes by bringing Module 3’s containerized CityFlow ETL job — the one that read the real January 2024 trip data and produced a verified zone-hour report — into a proper Compose stack: a real Postgres HEALTHCHECK, depends_on with a genuine readiness condition instead of a fixed sleep, and the report written into the database itself rather than only printed to a terminal.
Start with Lesson 1, and feel the difference between five commands that all have to agree, and one file that says it once.
Lessons in this module
1
Why Compose
Run the same CityFlow database-plus-reporter stack two ways: five hand-typed docker commands, then one Compose file and two commands.
2
Volumes
Remove and recreate a real Postgres container twice: once with no volume, losing everything, once with a named volume, keeping everything.
3
Networks
Inspect the network Compose has created automatically since Lesson 1, and prove a client container resolves 'db' by name with zero manual setup.
4
A Local Data Stack
Postgres and a real Python job, both defined as Compose services, aggregating 200,000 real CityFlow trips and loading the result into the database.
5
Guided Project: ETL and Postgres
Module 3's containerized CityFlow ETL job, now a Compose service that waits on a real Postgres healthcheck and writes its report into the database.
Achievement
Complete all 5 lessons to finish the Docker Compose module.
