Title here
Summary here
Module · 5 lessons
From Sequences to Attention
Why the transformer exists at all: the bottleneck recurrent networks hit on long sequences, and the idea — attention as a content-based weighted lookup — that replaced them. Build the attention mechanism's intuition in NumPy before formalizing it.
At a glance
Level
Advanced
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to From Sequences to Attention, the opening module. Before you build a transformer, you should understand the problem it was invented to solve — and feel, in real code, why the architecture that came before it wasn’t enough. That’s what this module is for.
You’ll start with the limits of recurrence: an RNN reads a sequence one step at a time, squeezing everything it has seen into a single fixed-size hidden state, and the signal from early tokens fades as the sequence grows. Then you’ll meet the fix — attention — not as an equation to memorize but as an idea you can build: a content-based weighted lookup, where each position pulls information from every other position, weighted by how relevant they are. You’ll implement that lookup in NumPy, learn how dot-product similarity and softmax turn raw scores into attention weights, add the learnable query/key/value projections that make it trainable, and finish by assembling a first retrieval-style attention layer over the course’s Lantern Bay corpus.
Start with Lesson 1, where you’ll watch a recurrent network struggle to hold onto what it read a moment ago.
Lessons in this module
1
The Limits of Recurrence
See exactly where recurrent networks break down on sequences — sequential computation, a fixed-size memory bottleneck, and fading long-range signal — with a real NumPy RNN.
2
The Attention Idea: Weighted Lookup
Learn attention as a soft, content-based lookup: compare a query to every key, turn the similarities into weights that sum to 1, and return a weighted average of the values.
3
Similarity, Scores & Softmax Weights
How attention turns query-key dot products into softmax weights: dot-product similarity, the numerically stable softmax, and temperature scaling — built from scratch in NumPy.
4
From Lookup to Learned Attention
Turn the raw query/key/value lookup into learnable attention by projecting the input through weight matrices Wq, Wk, and Wv in NumPy.
5
Guided Project: A Retrieval-Style Attention Layer
Assemble Module 1 into one runnable NumPy attention layer over the Lantern Bay corpus: tokenize, embed, run projected self-attention, and inspect the weight matrix.
Achievement
Complete all 5 lessons to finish the From Sequences to Attention module.
