SparkSession, RDDs & Lazy Evaluation

Welcome to SparkSession, RDDs & Lazy Evaluation, the second module — and the explanation Module 1 owed you. Twice in that module something strange happened and was deferred: reading the entire 160 MB quarter “completed” in about fifty milliseconds, while a mere count of the same data took half a second and spawned jobs, stages, and tasks. A pandas user’s instincts run exactly the other way — reading is the expensive part, counting is instant. The inversion has one cause, and it is the most important idea in Spark: nothing runs until an action demands a result. A line like df.filter(...) doesn’t filter anything; it appends a step to a plan. Spark is not an eager library that executes your lines — it is a compiler that collects them, optimizes the whole program, and executes it only when you ask for an answer.

That single design choice is where almost everything you’ll use in this course comes from. Because Spark sees the whole plan before running it, the optimizer can prune nineteen columns to three and push filters into the file scan — Module 1 watched it do both, unasked. Because a DataFrame is immutable and remembers the lineage of transformations that produced it, Spark can recompute a lost piece of work from source instead of replicating everything — its fault-tolerance story on a real cluster. And because the plan compiles down to RDDs — the raw distributed collections underneath — you can drop a level and see exactly what the DataFrame API is saving you, in measured seconds, by keeping your logic where the optimizer can see it.

Across five lessons you’ll time both halves of the lazy bargain, hand-roll a job on RDDs and measure it against Catalyst, learn to read the DAG — one action becoming jobs, stages split at shuffles, one task per partition, with Spark 4’s adaptive execution reshaping the plan mid-flight — and close with a guided project that traces a single real aggregation end to end, predictions written down before the run and checked against what the engine actually did. Start with Lesson 1, where the same pipeline is timed twice: once as a plan, once as work.