https://datatweets.com/courses/scaling-python-for-data-engineering/pandas-at-scale/Recent content in Pandas at Scale on DATATWEETSHugoenCopyright (c) 2026 DatatweetsMon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/pandas-at-scale/lesson-1-downcasting-numerics/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/pandas-at-scale/lesson-1-downcasting-numerics/Module 1 diagnosed the taxi data; this lesson delivers the first real fix. You’ll use pd.to_numeric(downcast=…) and astype to narrow the integer IDs to int16/int8 and the money columns to float32, prove downcasting is lossless for integers and bounded for floats, understand why to_numeric refuses to downcast trip_distance, and measure the whole DataFrame fall from 22.5 MB to 15.8 MB — every number taken live on the 200,000-row sample.https://datatweets.com/courses/scaling-python-for-data-engineering/pandas-at-scale/lesson-2-categoricals/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/pandas-at-scale/lesson-2-categoricals/The category dtype stores each distinct value once and gives every row a small integer code. On the real NYC taxi sample that cuts payment_type by 8x and a joined pickup-borough name column by nearly 17x. This lesson measures those wins live, shows the opposite case where a near-unique column costs more as a category, and previews why group-by on categoricals needs observed=True.https://datatweets.com/courses/scaling-python-for-data-engineering/pandas-at-scale/lesson-3-efficient-operations-and-copies/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/pandas-at-scale/lesson-3-efficient-operations-and-copies/Many pandas operations return a fresh copy, and stacking copies is how a pipeline’s memory quietly doubles or triples in the middle of a run. This lesson shows which operations copy, demonstrates the chained-indexing trap honestly under pandas 3.0’s Copy-on-Write (the assignment raises ChainedAssignmentError and never reaches the original frame) and its correct single-.loc fix, then rebuilds a wasteful six-copy taxi pipeline as one lean method chain — measured at 106 MB versus 40 MB peak for a byte-identical result.https://datatweets.com/courses/scaling-python-for-data-engineering/pandas-at-scale/lesson-4-fast-groupby-and-aggregation/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/pandas-at-scale/lesson-4-fast-groupby-and-aggregation/Group-by is the workhorse of analytics, and this lesson makes it fast on 200,000 real NYC taxi trips. You’ll write clean named aggregations to get revenue and average fare by payment type and trips by hour of day, then see how observed=True on categorical keys collapses a 58,985-row Cartesian blow-up down to the 9,371 groups that actually exist. You’ll measure a .groupby().apply() against the equivalent .agg() and clock an ~11x speedup for identical numbers, and finish with transform to subtract each zone’s mean fare from every trip in one pass.https://datatweets.com/courses/scaling-python-for-data-engineering/pandas-at-scale/lesson-5-guided-project-memory-efficient-taxi-loader/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/pandas-at-scale/lesson-5-guided-project-memory-efficient-taxi-loader/The capstone of Module 3. You measure a naive load of the full 2,964,624-row month (398.6 MB), then package everything from the module into one reusable load_taxi() function: keep only the columns the dashboard needs, downcast integers and floats to the smallest safe dtype, and turn payment type into a categorical. Loading the same month lean cuts memory to 115.9 MB — a measured 3.4x reduction, 70.9% saved, with every row intact — and you close by running a real revenue-by-payment-type analysis on the lean frame.