Title here
Summary here
Module · 5 lessons
When Data Outgrows Memory
The moment your data stops fitting: measure exactly why a 50 MB taxi file needs over 400 MB of RAM, learn to profile memory and dtypes, meet the data engineer's toolkit for taming it, and load only what you need — on the real NYC taxi dataset.
At a glance
Level
Beginner to Intermediate
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to When Data Outgrows Memory, the opening module of the course. Before you learn the techniques for handling large data, you should feel the problem precisely — in megabytes, on real data — so every fix that follows has a number attached to it.
You’ll start at the memory wall: load a month of real NYC taxi trips and watch a 50 MB file expand to over 400 MB of RAM, then understand exactly why. You’ll learn to profile a DataFrame’s memory column by column so you know where the weight is, survey the data engineer’s toolkit — the sequence of techniques this course teaches, from dtype tricks to chunking to distribution — and get your first concrete win by loading only the columns and rows you actually need. The module closes with a guided project that profiles the full dataset and turns the findings into a reduction plan you’ll execute later in the course.
Start with Lesson 1, and let the taxi data show you the wall.
Lessons in this module
1
The Memory Wall
Load one real month of NYC yellow-taxi trips and measure why a 48 MB file needs nearly 400 MB of RAM in pandas.
2
Profiling Memory & Dtypes
Measure exactly where a DataFrame's memory goes with df.info(deep) and memory_usage(deep=True), read the dtype cost table, and build a reusable profile_memory helper on the real NYC taxi sample.
3
The Data Engineer's Toolkit
A roadmap of the six techniques for handling data larger than memory, ordered from cheapest to heaviest, with a real before/after memory measurement on the NYC taxi sample.
4
Loading Only What You Need
Cut a DataFrame's memory at read time on the NYC taxi data using usecols, dtype, parse_dates, nrows, and parquet columns=.
5
Guided Project: Profile the NYC Taxi Data
Fully profile a real month of NYC taxi trips, find where the memory goes, and turn the findings into a measured reduction plan that cuts a 398.6 MB DataFrame to 115.9 MB.
Achievement
Complete all 5 lessons to finish the When Data Outgrows Memory module.
