Scaling Python for Data Engineering

What you'll build

You'll join CityFlow, a small mobility-analytics team, and learn to process data that's too big to fit comfortably in memory — using the real, public New York City yellow-taxi trip dataset (nearly three million trips a month). You'll measure why a 50 MB file balloons to over 400 MB in pandas, shrink it with the right dtypes, load only the columns you need, process it in chunks that never blow up your RAM, speed it up with parallel processing, and build the data structures — indexes, queues, trees — that real pipelines rely on. Every technique is measured for real on the actual dataset, so the numbers you see are the numbers you'll get.

Before you start

You'll need comfortable Python and a working knowledge of pandas basics — reading a CSV, selecting columns, grouping and aggregating. If you'd like a refresher first, our Python for Data Analytics course covers everything you need. No prior data-engineering or big-data experience is assumed; this course starts exactly where "my data got too big for pandas" begins.

Set up your environment

You can complete this course on any laptop with Python 3.10+. There's nothing to pay for and no cloud account — the dataset is a free, public download.

1. Install the packages the course uses:

pip install pandas numpy pyarrow

Every lesson fetches its data straight from a public URL — a curated sample of real trips for the quick examples, and the full monthly file from the New York City Taxi & Limousine Commission when it's time to feel the weight of real big data.