Welcome to What Is a Data Pipeline?, the opening module of Orchestrating Data Pipelines with Airflow. You’re back with CityFlow, the mobility-analytics team whose pipeline you’ve already made fast (Scaling Python), big (PySpark), and portable (Docker & Kubernetes). All three of those wins share a quiet assumption: someone starts the pipeline, in the right order, and notices if a step fails. So far, that someone has been you, sitting at a terminal. This module is about the moment that stops being good enough.

You’ll start concretely, not abstractly: CityFlow’s real ingest→validate→transform→load shape, defined in terms of tasks, the dependencies between them, why re-running a task safely (idempotency) matters more than it sounds like it should, and how scheduling turns a script into something that runs itself. Then you’ll actually build a small pipeline by hand — real Python scripts, chained the way cron would chain them — and watch it produce a real report from a real month of CityFlow’s taxi data. That success is deliberate: you need to feel the chain working before you’re in a position to feel it break.

And it does break, in specific, concrete ways you’ll reproduce yourself: a step that silently runs on stale or partial output because nothing tracks dependencies for you, a chain that dies outright on a transient failure with no retry, log files scattered across the filesystem with no single place to see what actually happened. Those are the exact gaps Apache Airflow was built to close, and once you’ve felt them, its core mental model — a DAG of tasks, run by a scheduler, an executor, and a webserver that don’t have to be the same process — will make immediate sense. The module closes with a guided project where you design a real DAG for a CityFlow job on paper, dependencies and all, before installing anything.

Start with Lesson 1, and define exactly what a data pipeline is before you decide it needs help.