Title here
Summary here
Module · 5 lessons
Positional Encoding & Embeddings
Attention has a blind spot: it can't tell word order. Learn why self-attention is permutation-invariant, build token embeddings from scratch, and add position information with both sinusoidal and learned positional encodings — assembling the input layer of a GPT in NumPy.
At a glance
Level
Advanced
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Positional Encoding & Embeddings, the fourth module — where you fix a blind spot you’ve already glimpsed. Back in Module 1, identical characters produced identical attention rows; that wasn’t a bug, it’s a fundamental property. Self-attention is permutation-invariant: it sees a sequence as an unordered set of vectors and has no idea which token came first. A language model that can’t tell “the boat” from “boat the” is useless, so we have to inject position deliberately.
You’ll start by demonstrating that invariance in NumPy, then build the two halves of a transformer’s input. First, token embeddings — a learned lookup table that turns character ids into vectors, with its forward pass and the scatter-add backward pass that trains it. Then position: the original sinusoidal encoding with its elegant sin/cos construction, and the learned positional embeddings that GPT actually uses, compared side by side. The guided project combines them into the complete input layer — embed the tokens, add their positions — that every later module feeds forward from.
Start with Lesson 1, where shuffling the input leaves attention completely unmoved.
Lessons in this module
1
Why Attention Needs Position
See attention's fundamental blind spot: self-attention is permutation-invariant, so shuffling the input just shuffles the output. Prove it in NumPy, then add position to break the symmetry.
2
Token Embeddings
Build the embedding table that turns token ids into learned vectors: a row lookup forward, a scatter-add backward, verified with a numerical gradient check in NumPy.
3
Sinusoidal Positional Encoding
Build the original Transformer's fixed sinusoidal positional encoding from scratch in NumPy and see why it gives every token a unique, bounded, extrapolatable sense of position.
4
Learned Positional Embeddings
Build the learned positional embedding table GPT actually uses: a trainable P of shape (block_size, C) you add to token embeddings, with a gradient check.
5
Guided Project: An Embedding Layer with Positions
Package token embeddings and learned positional embeddings into one reusable, gradient-checked NumPy EmbeddingLayer — the input stage of the GPT you build next.
Achievement
Complete all 5 lessons to finish the Positional Encoding & Embeddings module.
