Config, Storage and Batch Jobs

Welcome to Config, Storage and Batch Jobs, the sixth module of Docker & Kubernetes for Data Engineering. Module 5 gave CityFlow a real service running on Kubernetes — a Deployment and a Service, self-healing and reachable by name. But that service had everything hardcoded into its image, stored nothing that survived a pod being deleted, and only knew how to run forever. This module fills in every one of those gaps with the pieces a real data pipeline actually needs.

You’ll start by externalizing a service’s configuration two different ways — a ConfigMap for its ordinary settings, and a Secret for anything sensitive, with a close look at exactly why kubectl get secret -o yaml shows base64 rather than the real value, and why that is not encryption on its own. From there you’ll give Kubernetes its own answer to Module 4’s Docker volume: a PersistentVolumeClaim whose data survives a pod being deleted and recreated, backed by the StorageClass this cluster already runs. Then you’ll meet the workload shape a batch pipeline actually needs instead of an always-on Deployment — a Job that runs to real completion and reports Complete, with genuine retry behavior when it fails — and schedule that same shape to fire automatically and repeatedly with a CronJob, watched creating several real, successive runs on its own.

The module closes with a guided project that ties all four mechanisms together on one original, real batch task: a Job, configured entirely by a ConfigMap, that downloads and aggregates the real NYC taxi data this course has used since Course 1, with its output surviving on a PersistentVolumeClaim long after the Job’s own pod is gone.

Start with Lesson 1, and watch one small service behave two completely different ways with no image rebuild at all.