Building Images for Data Jobs

Welcome to Building Images for Data Jobs, the third module of Docker & Kubernetes for Data Engineering. Modules 1 and 2 taught you to run containers and to write a Dockerfile at all. This module is about writing a good one — the kind a data engineering team actually ships, where every choice has a measured reason instead of a guess.

You’ll start by swapping one line in a working Dockerfile — python:3.13 for python:3.13-slim — and measuring exactly how many hundreds of megabytes that costs or saves. From there you’ll hit a case the slim base alone can’t solve: a dependency that needs a real C compiler to install, and a multi-stage build that lets you keep the compiler for the build and throw it away before you ship. You’ll slim a bloated build context with .dockerignore and see, in real byte counts, why combining RUN instructions actually shrinks an image while splitting them doesn’t. And you’ll learn that “did it work?” means something different for a batch job than for a service that’s supposed to run forever — one reports through its exit code, the other through a HEALTHCHECK Docker keeps asking, and a restart policy is what turns a service’s failure into an automatic recovery instead of a 3 a.m. page.

The module closes with CityFlow’s first real Dockerfile — a containerized slice of the exact zone-hour ETL job Course 1 built in pandas, run against the real January 2024 trip data, mounted at run time rather than baked into the image, with every lesson’s technique applied and the real image-size win measured start to finish.

Start with Lesson 1, and find out what one line in a FROM instruction is actually worth.