Spark SQL & Execution Plans

Welcome to Spark SQL & Execution Plans, the fourth module — where you learn that the API you spent Module 3 mastering was never the real thing. Under df.filter(...) and df.groupBy(...) sits Catalyst, Spark’s query optimizer, and the DataFrame API is only one of two ways to feed it. The other is SQL. Register a DataFrame as a temp view and the exact zone-hour report you built last module becomes a SQL string — and it compiles to a byte-identical physical plan, because both front-ends meet in the same engine. Neither is faster; they are two dialects for describing the same work, and fluent data engineers move between them by which reads more clearly for the task.

Once you see that both roads lead to one plan, the plan itself becomes the thing worth reading — and .explain() becomes this module’s primary instrument. A physical plan is not mysterious once you can name its parts: a FileScan at the leaves that already knows which columns it needs and which filters to push down, HashAggregate nodes that pre-aggregate on each partition before a Exchange shuffles the small partial results, a final aggregate, and the AdaptiveSparkPlan wrapper that rewrites the whole thing at runtime. You’ll learn to read a plan bottom-up, point at any node and say what it does, and — crucially — see Catalyst’s optimizations as concrete before-and-after diffs: nineteen columns pruned to three, a filter pushed into the scan. You’ll also meet pushdown’s honest limit, because the taxi files ship no statistics, so a pushed filter skips no data on them — and you’ll build a file where it does, to see the difference.

Then the module draws the optimizer’s boundary. A Python UDF is a black box Catalyst cannot see through: it can’t prune, push, or generate code across it, and every row pays to cross into Python and back. You’ll measure exactly what that costs against a built-in expression and a vectorized pandas_udf, and learn the ordering that follows. The guided project puts it all together on a deliberately slow query — a UDF where a built-in would do, a needless wide select, a forced shuffle join, a filter applied too late — which you’ll diagnose from its plan and fix one measured step at a time, until it’s fast and still produces the verified answer. Start with Lesson 1, where the report you already wrote turns out to be a SQL query in disguise.