Title here
Summary here
Module · 5 lessons
Chunked & Out-of-Core Processing
Process data that never fully fits in RAM: stream a file in chunks with read_csv(chunksize=) and pyarrow batches, accumulate aggregates without holding the whole dataset, load chunks into a local SQLite warehouse, and build incremental, resumable pipelines — all on the real NYC taxi data.
At a glance
Level
Beginner to Intermediate
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Chunked & Out-of-Core Processing, the fourth module — where you stop needing the data to fit in memory at all. Modules 1 through 3 made a DataFrame as small as it can honestly be. This module handles the case where even the leanest frame is still bigger than your RAM: a full month of NYC taxi trips is nearly three million rows, and several months at once is out of the question for a naive read_csv.
The idea is simple and powerful: read the data in chunks, process each chunk, keep only a small running result, and discard the chunk. You’ll stream a CSV with read_csv(chunksize=) and a Parquet file with pyarrow batches, accumulate sums and counts to compute a mean over data you never fully hold, load chunks into a local SQLite warehouse you can query with real SQL and indexes, and make the whole thing incremental and resumable so appending next month’s data — or restarting after a crash — is safe and cheap.
Start with Lesson 1, where a file far too big for memory is processed with a footprint that never grows.
Lessons in this module
1
The Chunking Idea
Process a file too big for RAM by streaming it in fixed-size chunks so peak memory stays flat, using pd.read_csv(chunksize=...) and pyarrow's iter_batches on the real NYC taxi data.
2
Streaming Aggregation
Compute means, sums, counts, and group-by aggregates over data too big for RAM by accumulating partial (sum, count) results chunk by chunk, proven equal to the whole-file answer on the real NYC taxi data.
3
SQLite as a Local Warehouse
Stream a full month of NYC taxi trips into a local SQLite database with stdlib sqlite3, then run fast SQL GROUP BY and indexed WHERE queries over 2.96M on-disk rows without ever loading them all into pandas.
4
Incremental & Resumable Pipelines
Append each new month of NYC taxi trips to a SQLite warehouse without reprocessing old months, make re-runs idempotent with a ledger table so nothing is double-counted, and let a crashed load resume where it stopped.
5
Guided Project: Aggregate Months That Don't Fit
Build one end-to-end out-of-core pipeline that streams two full months of NYC taxi trips — 5,972,150 rows, far more than fits comfortably in RAM — into a local SQLite warehouse and rolls them up into a tidy per-zone, per-day summary with a peak memory footprint of under 10 MB.
Achievement
Complete all 5 lessons to finish the Chunked & Out-of-Core Processing module.
