Kubernetes Concepts on Kind

Welcome to Kubernetes Concepts on Kind, the fifth module of Docker & Kubernetes for Data Engineering. Modules 1 through 4 gave CityFlow’s pipeline everything Docker alone can offer: images, your own Dockerfiles, and a Compose file that starts a whole stack with one command, on one machine. This module answers the question Compose can’t: what happens when “one machine” isn’t enough — when a container needs to be rescheduled onto a different machine after a failure, or a fleet of services needs a control plane deciding where each one runs? That’s Kubernetes, and every example in this module runs on a real, already-running local cluster: Kind (Kubernetes in Docker).

You’ll start by proving, literally, that Kind is not a separate piece of infrastructure — it’s Kubernetes’ entire control plane running as containers inside one Docker container you already know how to inspect. From there you’ll run your first real pod by hand and watch its actual lifecycle (Pending → ContainerCreating → Running), then move to a Deployment: kill a pod on purpose and watch Kubernetes notice and replace it in seconds, then scale it to three replicas with one command. You’ll give a fleet of pods a stable network identity with a Service, inside its own Namespace, and see the real Kubernetes Dashboard confirm what kubectl already told you. The module closes with a guided project: an original CityFlow service — a small API over the real 265-zone NYC taxi lookup table — built, loaded into the cluster, deployed as a Deployment and Service, verified with both kubectl and the Dashboard, and torn down cleanly.

Every command in this module runs against datatweets-docker-k8s, a real Kind cluster already running on this machine — nothing here is simulated, and nothing you build outlives its own lesson.

Start with Lesson 1, and watch docker ps and kubectl disagree about how many things are running — for a very good reason.