Welcome to Airflow + Spark + Kubernetes, Module 7 of Orchestrating Data Pipelines with Airflow. Every prior module in this course ran a task’s own Python code directly, inside Airflow’s own workers. This module changes that shape for the first time: a task’s job is no longer to do the work, but to hand the work to Kubernetes — submit a real Job manifest, wait for a real pod to finish it, and read back a real result. It’s also where three separate courses in this path finally sit in the same DAG: Course 2’s PySpark, Course 3’s Kind cluster and Kubernetes Dashboard, and this course’s own Airflow orchestration.

No apache-airflow-providers-cncf-kubernetes is installed in this course’s Airflow image, and this module doesn’t install it — a heavier dependency than a learning environment needs, and not the only honest way to talk to Kubernetes from a task. Instead, every task that needs Kubernetes shells out, via subprocess, to a real kubectl binary and a real kubeconfig this course host-mounted directly into the Airflow containers, pointed at datatweets-docker-k8s — the same Kind cluster Course 3 built, reused here rather than recreated. That kubeconfig authenticates as the admin-user ServiceAccount over https://host.docker.internal:<port> with insecure-skip-tls-verify: true, a real, stated local-dev trade-off: Kind’s own TLS certificate simply has no host.docker.internal entry in its list of valid names.

Lesson 1 lays out the real shape end to end before any code runs: an Airflow task builds a Job manifest and applies it via kubectl; Kubernetes schedules a real pod running the PySpark image; the pod does real work and exits; a second Airflow task polls the Job’s own status field until it reports success. Lesson 2 proves that submission-and-polling mechanism with the simplest possible real pod — python:3.13-slim printing one line and exiting 0 — triggered for real through Airflow and confirmed with real kubectl get pods and kubectl logs output. Lesson 3 replaces that trivial pod with the real thing: a 869 MB image built from python:3.13-slim plus a headless JDK 21 JRE plus pyspark, with a real 81,013-row CityFlow slice (January 1st, 2024) baked directly into the image, doing a real Spark join-and-groupBy over real trip and revenue data. Lesson 4 watches one real run of that job through two completely independent systems at once: the Airflow Grid view and the Kubernetes Dashboard’s own Jobs page, the same real event seen from two different angles. The guided project closes the module by combining submission, polling, and verification into one clean DAG, triggered end to end and checked against Lesson 3’s own already-verified numbers, to the row.

Start with Lesson 1, and see the whole real path a task takes from an Airflow scheduler to a Kubernetes pod and back — before building a single piece of it.