Lesson 4 - Meet Airflow, Conceptually

Welcome to Meet Airflow, Conceptually#

Lesson 3 named four real gaps in a hand-built pipeline. Apache Airflow exists to close exactly those four gaps — this lesson builds the mental model it uses to do that, entirely on paper. No Docker, no install, nothing running: just the ideas, so that Module 2’s actual install lands on a mind already prepared for what it’s looking at.

By the end of this lesson, you will be able to:

• Define a DAG precisely: a directed graph of tasks with no cycles
• Explain why “acyclic” isn’t a technicality — what a cycle would actually mean for a pipeline
• Represent a DAG as plain data (a dict of task → dependencies) and validate it in real code
• Explain what the scheduler, executor, and webserver each do, and why splitting them up fixes Lesson 3’s gaps

A DAG Is Just: Tasks, and What Each One Needs First#

DAG stands for Directed Acyclic Graph. Unpacked:

• Graph — a set of things (nodes, here tasks) connected by relationships (edges, here dependencies).
• Directed — each edge has a direction: “transform depends on validate” is not the same statement as “validate depends on transform.”
• Acyclic — following the arrows can never lead back to where you started.

That last property is the one worth sitting with. Lesson 1’s dependency was ingest → validate → transform → load. Encode that exact chain as data — literally a dictionary mapping each task to the tasks it depends on — and write a small function that turns it into a valid run order:

def topological_order(dag):
    """Return a valid run order for a DAG expressed as {task: [upstream_deps]}.
    Raises ValueError if the graph has a cycle (i.e. isn't actually a DAG)."""
    remaining = {task: set(deps) for task, deps in dag.items()}
    order = []
    while remaining:
        ready = [t for t, deps in remaining.items() if not deps]
        if not ready:
            raise ValueError(f"cycle detected among: {sorted(remaining)}")
        ready.sort()
        for t in ready:
            order.append(t)
            del remaining[t]
        for deps in remaining.values():
            deps.difference_update(ready)
    return order

# CityFlow's ingest -> validate -> transform -> load pipeline, as data
cityflow_dag = {
    "ingest": [],
    "validate": ["ingest"],
    "transform": ["validate"],
    "load": ["transform"],
}

print("run order:", topological_order(cityflow_dag))

run order: ['ingest', 'validate', 'transform', 'load']

Nothing magic happened — the function repeatedly picks off tasks with no remaining unmet dependencies, which for a simple linear chain just reproduces the order you’d expect. The DAG earns its keep once dependencies stop being a straight line, which the guided project takes further. First, see what “acyclic” is actually protecting you from:

broken_dag = dict(cityflow_dag)
broken_dag["transform"] = ["validate", "load"]  # transform now (wrongly) depends on load
broken_dag["load"] = ["transform"]              # and load depends on transform: a cycle

try:
    topological_order(broken_dag)
    print("no cycle detected (unexpected)")
except ValueError as e:
    print("caught the cycle:", e)

caught the cycle: cycle detected among: ['load', 'transform']

transform now depends on load, and load depends on transform. Ask the honest question: which one would run first? There’s no answer — each is waiting on the other, forever. That’s not a style problem, it’s a pipeline that can never start. Airflow validates every DAG for exactly this before it ever runs a single task, refusing to schedule anything until the graph is provably acyclic. The function above does the same check in miniature: it can only make progress by finding a task with zero remaining dependencies, and a cycle guarantees no such task ever exists.

The Scheduler, the Executor, and the Webserver#

A DAG is the shape of the work. Something still has to decide when to run it, something has to actually run it, and something has to let a human see what happened. Airflow splits those three jobs into three different components instead of bolting them all into one script, the way Lesson 3’s hand-built chain did:

• The scheduler watches every DAG’s schedule and dependency state, and decides which tasks are ready to run right now. It’s the piece that replaces cron’s blind “just start this at 2 AM” with “start this at 2 AM, and keep checking whether each task’s upstream dependencies have actually finished successfully” — closing Lesson 3’s dependency-tracking gap directly.
• The executor is what actually runs a task once the scheduler says it’s ready — and, critically, it’s the piece that knows how to retry a task that failed, using whatever retry policy that task was given. This is where Lesson 3’s missing retry lives.
• The webserver is the UI — one place to see every DAG, every run, every task’s status and logs, replacing Lesson 3’s scattered step1.log/step2.log files with a single dashboard that always knows what ran and what didn’t.

All three read from and write to the same metadata database — the shared record of every DAG, every run, and every task’s state, which is what lets the scheduler, executor, and webserver stay in sync without talking to each other directly. Module 2 installs all four pieces for real and watches them talk to each other; for now, the important shift is that “did this pipeline run correctly” stops being a question only you can answer by remembering what you typed, and becomes a question the system itself can answer, because it kept the record.

Key Takeaways#

• A DAG is a Directed Acyclic Graph of tasks: edges have direction, and following them can never lead back to where you started.
• A DAG can branch (one task feeding two downstream tasks, or two tasks feeding one) without becoming invalid; only an actual cycle is forbidden.
• Representing a DAG as plain data — {task: [upstream_deps]} — is enough to validate it in real code: topological_order produced a correct run order for CityFlow’s chain and correctly raised on a deliberately broken, cyclic version.
• Airflow splits the work Lesson 3’s single script tried to do alone into a scheduler (decides what’s ready), an executor (runs it, retries it), and a webserver (shows it to a human) — all three sharing one metadata database instead of keeping separate, scattered state.

Exercises#

Exercise 1 — Add a fifth task, notify, to cityflow_dag that depends on load. Run it through topological_order and confirm notify comes out last.

Exercise 2 — Build a small branching DAG by hand: ingest with no dependencies, then two independent tasks validate_fares and validate_zones that both depend only on ingest, then a combine task that depends on both. Run it through topological_order and confirm ingest comes first, combine comes last, and the two validate_* tasks appear somewhere in between (in either order relative to each other).

Exercise 3 — Introduce a self-cycle: make a single task depend on itself (e.g. dag["ingest"] = ["ingest"]). Confirm topological_order still raises ValueError, and explain in a comment why a task depending on itself is a special case of the same cycle rule, not a separate bug.