Why Distributed?

Welcome to Why Distributed?, the first module — which starts by taking the other side of an argument this path just won. The previous course proved a laptop can process ten million trips under a strict memory budget, and that “too big for pandas” is usually a solvable problem, not a wall. All of that remains true. And yet every serious data team eventually runs Spark, Snowflake, BigQuery, or something like them — engines built to spread work across many machines. This module is about why, honestly: not because one machine is weak, but because it is one — one set of cores that parallelism can saturate but never exceed, one disk with one throughput, one memory ceiling that streaming defers but never removes, and one failure domain for a job that must finish tonight.

Apache Spark is the tool this course uses to cross that line, and its best teaching feature is that you don’t need a cluster to learn it truthfully. Spark’s local mode runs the genuine article — the same driver, the same optimizer, the same DataFrame API, the same execution plans — with your CPU cores standing in for a cluster’s executors. Code written this way moves to a real cluster by changing one connection string, which means everything you measure here is real Spark behaviour, not a simulation of it.

Across five lessons you’ll find the ceiling with real measurements, learn Spark’s architecture well enough to predict what it does before it does it, start your first SparkSession over CityFlow’s full taxi quarter, and run an honest pandas-versus-Spark comparison — one that includes the JVM startup cost and shows exactly where Spark is the wrong tool. The guided project ends with CityFlow’s first Spark artifact: the whole quarter loaded, profiled, and sanity-checked against the numbers the previous course established. Start with Lesson 1, where the machine that just handled ten million trips meets the limits no technique can optimize away.