DataFrames & Transformations

Welcome to DataFrames & Transformations, the third module — where you stop watching Spark and start driving it. The first two modules were about understanding: the ceiling that makes distributed engines necessary, and the machinery underneath Spark — lazy plans, RDDs, the DAG, lineage. You can now predict what Spark will do. This module makes you fluent enough to tell it what to do, correctly, on data that fights back. The DataFrame API is the tool you’ll use every working day, and it rewards knowing it well: expressed cleanly, a transformation is something Catalyst can optimize; expressed carelessly, it’s a shuffle you didn’t need or a filter that quietly corrupts a total.

The through-line is CityFlow’s real taxi quarter, and the data’s genuine defects are the point rather than a distraction. You’ll build derived columns — trip duration, fare-per-mile, pickup hour — with column expressions that are themselves lazy plan nodes. You’ll filter and clean in Spark, and the central lesson there is restraint: the dataset’s tens of thousands of negative fares are accounting reversals, not errors, and a filter that drops them overstates revenue by a measured six figures a month. You’ll join the 265-row zone dimension with a broadcast that skips the shuffle entirely, and you’ll key that join on the numeric ID because two different zones are both named “Corona.” And you’ll group and aggregate per zone and per hour, the wide transformation the whole report is built on.

Then the module pays off everything before it. The guided project rebuilds Course 1’s entire zone-hour report — the capstone of the previous course, which pandas and multiprocessing produced under a memory budget — in idiomatic Spark, and cross-verifies it to the cent: 240,917 buckets, 9,554,757 trips, $256,692,373.14. Same answer, same real data, a cluster-scale tool. Start with Lesson 1, where a column turns out to be not a list of values but an expression waiting to be planned.