Lesson 5 - Guided Project: Design a DAG on Paper

Welcome to the Guided Project#

Every lesson in this module built toward one skill: looking at a real job and seeing its task list and dependencies before writing a line of orchestration code. This project is that skill, applied for real. CityFlow wants a richer daily report than Lesson 2’s — one that breaks revenue down by both fare tier and pickup zone — and your job is to design its DAG on paper: a task list, a dependency table, a diagram, and then a real correctness check, with still no Airflow installed.

By the end of this project, you will be able to:

• Break a real job into individually meaningful tasks, not just “steps that felt convenient to split”
• Identify which tasks are true dependencies of each other and which can run independently
• Represent that design as a dependency table and a diagram, and confirm the two agree
• Validate a hand-designed DAG with real code before ever handing it to an orchestrator

The Job: CityFlow’s Zone-and-Fare Daily Report#

CityFlow wants one report per day that answers two questions at once: “how did fares break down?” and “how did pickup zones break down?” Both questions start from the same validated trips, and both feed into one final combined report. Written out as a task list:

1. ingest_taxi_data — pull the day’s raw trips from the source parquet.
2. validate_trips — drop rows that can’t be real trips (the same rules Lesson 1 used).
3. compute_fare_summary — group validated trips by fare tier, compute counts and revenue per tier.
4. compute_zone_summary — group validated trips by pickup zone, compute counts and revenue per zone.
5. merge_summaries — combine the fare summary and the zone summary into one report object.
6. load_report — write the final combined report out.

The Dependency Table#

Design decisions live in this table, not in prose — this is the artifact a real orchestrator would eventually read:

The load-bearing design decision is in the two middle rows: compute_fare_summary and compute_zone_summary do not depend on each other. Each needs only validate_trips’s output, so they’re genuinely independent — an orchestrator could run them at the same time, on different workers, and Lesson 3’s hand-built && chain never could have expressed that, because && only knows how to run things one after another.

The Diagram#

The same design, drawn as a graph:

Notice the diagram and the table say exactly the same thing, just in two different notations. That agreement is not a coincidence you should hope for — it’s something you should be able to check.

Validating the Design with Real Code#

A dependency table is still just prose with structure. Encode it as the same {task: [upstream_deps]} data shape from Lesson 4, and run it through the same validator:

def topological_order(dag):
    remaining = {task: set(deps) for task, deps in dag.items()}
    order = []
    while remaining:
        ready = [t for t, deps in remaining.items() if not deps]
        if not ready:
            raise ValueError(f"cycle detected among: {sorted(remaining)}")
        ready.sort()
        for t in ready:
            order.append(t)
            del remaining[t]
        for deps in remaining.values():
            deps.difference_update(ready)
    return order

# The guided project's design, encoded exactly as drawn above:
cityflow_daily_report_dag = {
    "ingest_taxi_data": [],
    "validate_trips": ["ingest_taxi_data"],
    "compute_fare_summary": ["validate_trips"],
    "compute_zone_summary": ["validate_trips"],
    "merge_summaries": ["compute_fare_summary", "compute_zone_summary"],
    "load_report": ["merge_summaries"],
}

order = topological_order(cityflow_daily_report_dag)
print("a valid run order:", order)

a valid run order: ['ingest_taxi_data', 'validate_trips', 'compute_fare_summary', 'compute_zone_summary', 'merge_summaries', 'load_report']

The function found a valid order without being told one — proof the design is genuinely acyclic, not just acyclic-looking on a whiteboard. Two more checks make the design trustworthy rather than just plausible:

# Every dependency must point at a task that actually exists in the design --
# a real mistake to catch on paper, before it becomes a broken Airflow DAG.
all_tasks = set(cityflow_daily_report_dag)
for task, deps in cityflow_daily_report_dag.items():
    for dep in deps:
        assert dep in all_tasks, f"{task} depends on undefined task {dep}"
print("every dependency resolves to a real task in the design: ok")

# Confirm the two summary branches are genuinely independent of each other --
# the exact parallelism an orchestrator can exploit that a linear && chain cannot.
i_fare = order.index("compute_fare_summary")
i_zone = order.index("compute_zone_summary")
i_validate = order.index("validate_trips")
i_merge = order.index("merge_summaries")
print("validate_trips runs before both branches:", i_validate < i_fare and i_validate < i_zone)
print("merge_summaries runs after both branches:", i_merge > i_fare and i_merge > i_zone)

every dependency resolves to a real task in the design: ok
validate_trips runs before both branches: True
merge_summaries runs after both branches: True

Every dependency in the table points at a task that’s actually defined — a real, easy-to-make mistake (a typo in a task name) that this check catches before it ever reaches an orchestrator. And the ordering confirms the design decision that mattered most: validate_trips genuinely precedes both summary tasks, and merge_summaries genuinely follows both, with nothing forcing an order between the two summary tasks themselves. That’s the whole point of drawing this as a DAG instead of a list: the design captures exactly which steps must wait for which, and no more than that.

What This Project Actually Proved#

Trace this back to the module. Lesson 1 gave you the vocabulary — tasks, dependencies, idempotency, scheduling. Lesson 2 proved a hand-built chain can do real, correct work end to end. Lesson 3 showed exactly where that chain breaks: silent staleness, no retry, scattered logs, no backfill. Lesson 4 gave you the DAG mental model and the scheduler/executor/webserver split that fixes those breaks. This project is all four ideas at once, applied to a job real enough that its two branches — fare summary and zone summary — are actually independent of each other, not independent by accident. You designed it, drew it two different ways, and then proved with real code that both descriptions agree and the result is a valid, acyclic, fully-resolved DAG — before Apache Airflow has been installed anywhere.

Key Takeaways#

• A real DAG design should exist as more than one artifact — a dependency table and a diagram — and the two should be checked against each other, not just against your intuition.
• Two tasks belong in the same “branch” (no edge between them) when each depends only on a shared ancestor, not on each other — here, compute_fare_summary and compute_zone_summary both depend only on validate_trips.
• Encoding a design as {task: [upstream_deps]} data lets you validate it with real code: acyclic (a run order exists), fully resolved (every dependency points at a real task), and correctly parallel (independent tasks have no artificial ordering between them).
• This is the artifact Module 2 hands directly to Airflow — the TaskFlow API and operators you’ll meet there are just a way of expressing exactly this same {task: [upstream_deps]} shape in a form the scheduler can execute.

Exercises#

Exercise 1 — Design your own DAG on paper for Lesson 3’s cron-demo job (step1_extract_today, step2_enrich_zone_reference, step3_aggregate, step4_report) as a dependency table. Encode it as a dict and confirm topological_order reproduces the same linear order the && chain used.

Exercise 2 — Add a seventh task, notify_team, to this lesson’s cityflow_daily_report_dag that should run after load_report and independently log a “started” message that should run immediately after ingest_taxi_data, before validation even begins. (Hint: this requires notify_team to depend on two different upstream tasks that are nowhere near each other in the graph — is that allowed by the acyclic rule? Confirm with topological_order.)

Exercise 3 — Deliberately design a broken version of this lesson’s DAG where compute_fare_summary is wrongly listed as depending on compute_zone_summary (instead of only on validate_trips). Run it through topological_order — does it still produce a valid order? Explain, using the dependency-table description of a “true” dependency from this lesson, why this version runs successfully but is still a design mistake, not a code bug.