Welcome to Sensors, Hooks & Connections, Module 4 of Orchestrating Data Pipelines with Airflow. Every DAG so far has run its tasks against files already sitting on disk when the DAG was triggered. Real pipelines don’t get that luxury — CityFlow’s actual mobility data doesn’t arrive on a schedule anyone controls, and it doesn’t live inside Airflow’s own containers. This module is about the two mechanisms Airflow gives you for exactly that gap: sensors, which wait for a real external condition instead of assuming a fixed delay, and hooks, which talk to a real outside system — a database, here — through a well-defined interface instead of hand-rolled connection code scattered across tasks.

You’ll start with a real FileSensor, triggered against a file that genuinely does not exist yet — its log shows real repeated Poking for file lines while a second process, running concurrently, creates the file mid-run. From there you’ll add a second live service to this course’s stack: cityflow-lessons-db, a real Postgres 13 container reachable from both your own machine and from inside the Airflow containers, talked to through a real PostgresHook — never raw psycopg2 — for a real CREATE TABLE, INSERT, and SELECT. Lesson 3 replaces every hardcoded piece of that setup with a real Airflow Connection and a real Airflow Variable, configured through the live UI and CLI and confirmed with real screenshots. Lesson 4 combines both ideas in one DAG — wait, then load — and the guided project runs that same pattern against 500 real CityFlow trip rows, verified with a real row count and a real Grid/Graph screenshot of the finished run.

Every DAG in this module is a real file, saved to the same .airflow-local/dags/ folder Modules 2-3 used, triggered against the same live Airflow 2.10.4 stack, and talking to a real second Postgres instance stood up just for this module — nothing here is a diagram of what a sensor or a hook would do.

Start with Lesson 1, and watch a real sensor wait for a file that isn’t there yet.