Title here
Summary here
Module · 5 lessons
Data Structures for Data Work
The right data structure makes the access pattern cheap. Learn why a dict finds a zone instantly while a list hunts for it, why popping the front of a list is a hidden trap and a deque is not, how a heap answers top-N without sorting anything, and how to choose between them by access pattern — all measured on the real NYC taxi data.
At a glance
Level
Beginner to Intermediate
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Data Structures for Data Work, the sixth module — where your pipeline stops doing work it never needed to do. Module 5 spread the job across every core on the machine and cut the wall clock by nearly four. That is a real win, but it is also a blunt one: parallelism makes slow code finish sooner without making it less wasteful. If every trip triggers a linear scan through a list of zones to find a borough name, six processes just perform that wasteful scan six times at once. The deeper fix is to change the shape of the data so the scan never happens.
That is what a data structure buys you. A dict finds a zone by id in the same time whether it holds ten entries or ten million, because hashing jumps straight to the answer instead of looking through the candidates. A set answers “have I seen this trip before?” the same way. A deque gives a streaming buffer a genuinely constant-time front, where a plain list quietly re-copies every element on each pop(0). A heap hands you the ten busiest zones without ever sorting the other two hundred and fifty-five. Each is a different bargain between memory, insertion cost, and lookup cost, and knowing which bargain fits the access pattern is the difference between a pipeline that scales and one that merely runs.
Across five lessons you’ll measure every one of those claims on CityFlow’s real taxi data, then assemble them: the guided project builds an in-memory zone index and a top-N ranking in a single pass over a month of trips. Start with Lesson 1, where a lookup that a list takes milliseconds to perform is answered by a dict in well under a microsecond — and the gap only widens as the data grows.
Lessons in this module
1
Dicts & Sets: O(1) Lookups & Dedup
Replace a list scan with a dict on CityFlow's zone lookup and measure the difference on real NYC taxi data — 36.5x faster on isolated lookups, 19.2x faster enriching 2,964,624 real trips — then dedup the same month with a set in 0.52 s where a list-based check would take about 16 hours.
2
Stacks & Queues
Give CityFlow's streaming pipeline somewhere to hold work that has arrived but isn't processed yet, and learn why the obvious choice is a trap: draining a 200,000-element queue takes 3,012.6 ms with list.pop(0) but 3.6 ms with collections.deque.popleft() - a measured 829x gap.
3
Heaps & Priority Queues
Answer CityFlow's top-10 questions without sorting everything else, using heapq on a real month of NYC taxi trips — 18.2x faster than a full sort on 2.96 million rows, and 30x leaner when the heap runs over a stream.
4
Choosing the Right Structure
Turn dicts, sets, deques, and heaps into a decision procedure by measuring all five structures across seven pipeline operations on 2.96 million real NYC taxi trips — and find the honest crossover point where building a dict index stops costing more than it saves: about 5 questions.
5
Guided Project: An In-Memory Index + Top-N
Build CityFlow's zone-activity service: one streaming pass over 2,964,624 real January 2024 taxi trips that fills a dict index, a bounded top-10 heap, and a set filter at once — 134 MB peak against the naive baseline's 1,026 MB, and the answers it keeps are 39 KB.
Achievement
Complete all 5 lessons to finish the Data Structures for Data Work module.
