Title here
Summary here
Module · 5 lessons
Parallel Processing
Use every CPU core to speed up data work — and measure the real gain. Understand the GIL and why data pipelines use processes not threads, drive a multiprocessing Pool, apply the map-reduce pattern, process partitions in parallel without paying serialization costs, and watch speedup plateau exactly as Amdahl's law predicts — all on the real NYC taxi data.
At a glance
Level
Beginner to Intermediate
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Parallel Processing, the fifth module — where your pipeline stops using one CPU core and starts using all of them. Everything so far ran on a single core: correct, memory-lean, but leaving most of your machine idle. A modern laptop has eight or ten cores; a data-engineering job that ignores them is running at a fraction of its potential.
This module is honest about when parallelism helps. You’ll learn why Python’s Global Interpreter Lock means threads give almost no speedup for CPU-bound work, while separate processes genuinely run in parallel. You’ll drive a multiprocessing Pool, structure work as map-reduce (compute partial results per partition, then combine them), and sidestep the biggest trap — the cost of serializing large data to workers — by having each worker read its own partition. Along the way you’ll measure everything, because the speedup is never the number of cores: it plateaus, exactly as Amdahl’s law predicts.
Start with Lesson 1, where the same real computation runs flat across threads and nearly four times faster across processes.
Lessons in this module
1
Why Parallelism (and the GIL)
See why Python data pipelines parallelize CPU-bound work with processes, not threads, by running the same heavy computation on six NYC taxi partitions three ways — serial, threads, and a multiprocessing Pool — and measuring how the GIL keeps threads flat while processes run nearly four times faster.
2
multiprocessing & Process Pools
Drive a multiprocessing Pool to spread six NYC taxi row-group partitions across worker processes, then sweep the worker count from 1 to 10 and measure the real, sublinear speedup curve — about 1.83x at 2 workers, 2.58x at 4, and 4.05x at 6 — on ~6 million trips.
3
The MapReduce Pattern
Structure a parallel aggregation as map-reduce: compute a small per-zone partial for each of the six taxi partitions in parallel with a multiprocessing Pool, then combine the partials into the final per-zone mean, and prove the parallel answer is identical to the serial one.
4
Parallel Chunk Processing
Combine Module 4's chunking with Module 5's parallelism on the real NYC taxi partitions, and measure the cost that decides whether it pays off — serializing data to and from worker processes.
5
Guided Project: Parallel Aggregation, Measured
Parallelize Module 4's per-zone/day taxi summary as a map-reduce over 6 row-group partitions, time it from 1 to 8 worker processes, and use Amdahl's law to explain why 6 cores buy roughly 4.3x, not 6x.
Achievement
Complete all 5 lessons to finish the Parallel Processing module.
