PySpark for Data Engineering

What you'll build

You'll rejoin CityFlow, the mobility-analytics team from Scaling Python for Data Engineering, as its data outgrows what one machine should carry alone. Using Apache Spark in local mode — a real SparkSession on your own laptop, no cluster or cloud account required — you'll learn how Spark thinks (drivers, executors, lazy evaluation, the DAG), master DataFrames and Spark SQL on nearly ten million real NYC yellow-taxi trips, read execution plans to see the optimizer at work, tame partitions, shuffles, and caching where Spark performance actually lives, and finish with a production-shaped ETL pipeline with schemas, data-quality gates, quarantine, and structured logging. Every result is verified against the numbers the previous course established — same data, same answers, cluster-scale tools.

Before you start

You'll need solid Python and comfortable pandas, and you'll get the most out of this course after Scaling Python for Data Engineering — this course reuses its dataset, its running example, and its measured results as ground truth. No prior Spark or cluster experience is assumed.

Set up your environment

Everything runs locally with Python 3.10+ and a Java runtime (Spark runs on the JVM). There is nothing to pay for and no cluster to rent.

1. Install a JDK (17 or 21 — e.g. Adoptium Temurin), then
2. Install the packages the course uses:

pip install pyspark pandas pyarrow

Every lesson fetches its data from a public URL — the same real NYC Taxi & Limousine Commission trip files the previous course used, so your results are comparable end to end.