https://datatweets.com/courses/scaling-python-for-data-engineering/numpy-for-data-engineering/Recent content in NumPy for Data Engineering on DATATWEETSHugoenCopyright (c) 2026 DatatweetsMon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/numpy-for-data-engineering/lesson-1-why-numpy/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/numpy-for-data-engineering/lesson-1-why-numpy/Using 200,000 real total_amount values from the NYC yellow-taxi sample, you’ll measure the gap between a Python list and a NumPy array: the list needs 6.1 MB of RAM to the array’s 1.5 MB (4x), and a NumPy sum runs about 50x faster than the equivalent Python loop. Then you’ll see exactly why — a list stores full Python float objects reached through pointers, while an ndarray packs raw single-dtype values in one contiguous block the CPU can stream through a C-level loop.https://datatweets.com/courses/scaling-python-for-data-engineering/numpy-for-data-engineering/lesson-2-dtypes-precision-and-memory/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/numpy-for-data-engineering/lesson-2-dtypes-precision-and-memory/Every NumPy array has a dtype that fixes its byte cost and the range of values it can hold. This lesson maps the integer and float dtypes to their sizes and ranges, then picks the smallest safe type for real taxi columns by their actual range, measuring the megabytes saved. You will watch int8 silently corrupt 77.8% of the PULocationID column because 265 wraps past its 127 ceiling, learn why int16 is the safe minimum, see where float32 is plenty and where it is not, and build a helper that picks a safe integer dtype from a column’s min and max.https://datatweets.com/courses/scaling-python-for-data-engineering/numpy-for-data-engineering/lesson-3-vectorization/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/numpy-for-data-engineering/lesson-3-vectorization/You’ll compute trip duration in minutes for 200,000 real NYC taxi trips two ways: with a Python for-loop over rows, and with a single vectorized NumPy operation that subtracts two datetime64 arrays and converts the timedelta to minutes. Timed head to head, the vectorized version runs about 1,700x faster — 531 ms down to 0.31 ms — for an identical result. Then you’ll add the other vectorized moves a pipeline needs: element-wise arithmetic for speed, boolean masks to flag long trips, and np.where for a conditional column, ending with the mental shift from thinking in elements to thinking in whole arrays.https://datatweets.com/courses/scaling-python-for-data-engineering/numpy-for-data-engineering/lesson-4-broadcasting/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/numpy-for-data-engineering/lesson-4-broadcasting/Vectorization gave you fast whole-array math on matching shapes; broadcasting extends it to arrays of different shapes. You’ll learn the right-to-left shape rule, scale trip_distance from miles to kilometres with a scalar, compute fare_per_mile element-wise, and subtract per-column means to center a (200000, 4) array with a (4,) vector — proving NumPy stretches the small array virtually instead of copying it. You’ll also learn to read the broadcasting error so you can debug shape mismatches fast.https://datatweets.com/courses/scaling-python-for-data-engineering/numpy-for-data-engineering/lesson-5-guided-project-trip-metrics-with-numpy/Mon, 13 Jul 2026 00:00:00 +0200https://datatweets.com/courses/scaling-python-for-data-engineering/numpy-for-data-engineering/lesson-5-guided-project-trip-metrics-with-numpy/The capstone of Module 2. You load six taxi columns as NumPy arrays with safe dtypes, then compute four per-trip metrics fully vectorized — trip duration, average speed, fare per mile, and tip rate — guarding against the real zero-distance and zero-duration trips in the data. You time the vectorized version against an equivalent Python loop, measure a several-hundred-fold speedup, confirm the results match with np.allclose, and use the metrics to surface one real insight and a data-quality problem.