Capstone

Welcome to the Capstone, the eighth and final module — one project, and the argument the whole course has been making.

Go back to Module 1 for a moment. A 50 MB file expanded to roughly 400 MB the instant pandas touched it, and that was the entire problem: the data was small, the machine was large, and the naive approach still fell over. Everything since has been a response. Module 2 made the numbers smaller. Module 3 made pandas lean. Module 4 stopped loading the file at all and streamed it into a warehouse instead. Module 5 put every core to work. Module 6 replaced scans with the right container. Module 7 restored the order that hashing threw away, so a range could be answered without reading everything. Each module left one reusable piece behind, and each piece was measured rather than asserted.

This module stacks them. You’ll build CityFlow’s out-of-core analytics pipeline over 9,554,778 real trips spanning three months — profiling what the naive path would cost, streaming the data in batches, shrinking dtypes, aggregating in parallel across partitions, loading a SQLite warehouse that can be re-run without double-counting, clustering and indexing it, and finally answering the per-zone, per-hour trip and revenue questions the dashboard actually asks. Every stage is labelled with the module that earned it.

The harder half is the data itself, and this is where the project stops being a tidy exercise. This dataset is genuinely defective in ways that punish a careless pipeline: tens of thousands of repeated keys that are accounting reversals rather than duplicates — deduplicate them and you overstate revenue; timestamps from 2002 and 2009 sitting in a January file, few enough to ignore and destructive enough to ruin an index; zone names that are not unique, so joining on the wrong column silently loses trips. A pipeline that is fast and wrong is worth nothing. Start the project, and build one that is neither.