https://datatweets.com/courses/scaling-python-for-data-engineering/when-data-outgrows-memory/Recent content in When Data Outgrows Memory on DATATWEETSHugoenCopyright (c) 2026 DatatweetsMon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/when-data-outgrows-memory/lesson-1-the-memory-wall/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/when-data-outgrows-memory/lesson-1-the-memory-wall/You’ll load a full month of real NYC yellow-taxi data — 2.96 million trips — and measure with df.memory_usage(deep=True) that a 48 MB file expands to nearly 400 MB in pandas, about 8.4x. Then you’ll see exactly why: full-width int64 and float64 dtypes, and Python-object strings that carry huge per-value overhead. Finally you’ll multiply the blow-up by a few months and watch it cross a laptop’s RAM — the memory wall this course is built to get you over.https://datatweets.com/courses/scaling-python-for-data-engineering/when-data-outgrows-memory/lesson-2-profiling-memory-and-dtypes/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/when-data-outgrows-memory/lesson-2-profiling-memory-and-dtypes/Lesson 1 showed the taxi data hitting the memory wall. This lesson teaches you to measure the wall precisely: the difference between shallow and deep memory accounting, per-column profiling to find the heaviest columns, the byte-per-value cost of every common dtype, and a reusable profile_memory() helper that ranks columns by memory share — every number measured live on the 200,000-row NYC taxi sample.https://datatweets.com/courses/scaling-python-for-data-engineering/when-data-outgrows-memory/lesson-3-the-data-engineers-toolkit/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/when-data-outgrows-memory/lesson-3-the-data-engineers-toolkit/Meet the full toolkit this course teaches: load less, shrink in place, process in chunks, parallelize, use better data structures, and go distributed. See where each technique lives in the course and why order matters, then watch two cheap fixes cut a naive 23.6 MB taxi load down to 3.8 MB — a 6.2x reduction from column selection and dtypes alone.https://datatweets.com/courses/scaling-python-for-data-engineering/when-data-outgrows-memory/lesson-4-loading-only-what-you-need/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/when-data-outgrows-memory/lesson-4-loading-only-what-you-need/Your first cheap, concrete win against the memory wall: stop loading data you never use. On the real NYC taxi sample you’ll measure a naive read_csv against one that selects columns with usecols, sets small dtypes, parses dates, and peeks with nrows, cutting RAM from 22.5 MB to 4.0 MB. Then you’ll push column selection down into a parquet file with columns= and watch a 398 MB load shrink to 79 MB.https://datatweets.com/courses/scaling-python-for-data-engineering/when-data-outgrows-memory/lesson-5-guided-project-profile-the-taxi-data/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/when-data-outgrows-memory/lesson-5-guided-project-profile-the-taxi-data/The capstone of Module 1. You load the full month of real NYC yellow-taxi trips (2,964,624 rows), measure the disk-to-memory blow-up, profile every column, and spot the cheaper representations for the heaviest ones. Along the way you catch genuine dirty data — a pickup timestamp from 2002 and 37,448 negative fares — then assemble and apply a reduction plan that shrinks the DataFrame 3.4x, all measured on the real data.