Capstone

Welcome to the Capstone, the eighth and final module — one project, and the argument this whole course has been making.

Go back to Module 1. You had a laptop that had just proved, in the previous course, that one machine used well can process ten million trips under a strict memory budget. The honest question was why anyone would reach for Spark at all — and the honest answer, measured, was that at this scale they shouldn’t: tuned streaming pandas beat Spark on every size we tested. What Spark bought was different and it took seven modules to earn: code that describes what you want and lets an optimizer decide how, a plan you can read and tune, and a job whose shape survives the data outgrowing the machine it started on. Everything since Module 1 has been building the vocabulary to use that properly — lazy plans and the DAG, the DataFrame API and SQL as two faces of one engine, Catalyst’s optimizations and the opaque UDFs that defeat them, partitions and shuffles and when a cache actually pays, the production ETL shape, and the guardrails that let a job run unattended.

This module puts all of it into a single runnable file. You’ll build CityFlow’s production pipeline: a job that configures and tunes its own session, declares an explicit schema contract, validates the quarter it was handed, cleans it while quarantining violations with reasons rather than deleting them, transforms with a broadcast join and not a single UDF, validates its own output, refuses to publish if a critical rule fails, writes idempotently so a retry replaces partitions instead of doubling them, and finishes by emitting a structured run summary and an exit code a scheduler can act on. Every stage traces back to the module that taught it, and you’ll read the finished job’s execution plan and recognize every node in it.

Then you’ll prove the job rather than trust it, three ways. It reproduces the numbers you now know by heart — 240,917 buckets, 9,554,757 trips, $256,692,373.14 — computed here by a distributed engine and matching, to the cent, what pandas and multiprocessing produced in the previous course. Running it twice changes nothing. And pointed at a deliberately broken feed, it aborts before writing a single file, exits nonzero, and leaves the last good table exactly as it was. That is the difference between a script that works and a pipeline you can hand to a scheduler and go to sleep. Start the project — and finish the course.