Title here
Summary here
Module · 5 lessons
Multi-Head Attention
One attention head learns one kind of relationship; a transformer runs many in parallel. Learn why multiple heads help, how to split the model dimension into heads, run them in parallel and concatenate, and backprop through the whole thing — building a gradient-checked multi-head attention layer in NumPy.
At a glance
Level
Advanced
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Multi-Head Attention, the third module. A single attention head, however well trained, can only express one attention pattern per position — one notion of what to look for. Real transformers run several heads at once, each free to attend to a different kind of relationship, and then combine what they find. This module builds that from scratch.
You’ll start with the motivation: why one head isn’t enough, shown with two heads that learn to attend to genuinely different things. Then you’ll implement the mechanics — splitting the model dimension into heads of size , running scaled dot-product attention in parallel across all of them, and concatenating the results before a final output projection mixes them back together. Finally you’ll derive the backward pass through the split, the concatenation, and the projection — reusing the single-head gradients from Module 2 — and verify it against numerical gradients. The guided project assembles a clean, gradient-checked multi-head attention layer you’ll stack into the full transformer block next.
Start with Lesson 1, where a single head runs out of room.
Lessons in this module
1
Why One Head Isn't Enough
See why a single attention head can only express one relationship per position, and run two independent heads in NumPy to prove they attend to genuinely different things.
2
Splitting into Heads
Learn the reshape and transpose mechanics that turn one wide (C, C) projection into h parallel attention heads of size hs = C // h.
3
Parallel Attention & Concatenation
Assemble the full multi-head attention forward pass in NumPy: run scaled dot-product attention independently per head, concatenate the heads, and project with Wo.
4
The Multi-Head Backward Pass
Derive and implement the full backward pass through multi-head attention in NumPy, split and merge included, then prove every gradient with a numerical gradient check.
5
Guided Project: A Multi-Head Attention Layer
Package this module into one reusable, gradient-checked NumPy MultiHeadAttention class with Wq, Wk, Wv, Wo that you stack into the transformer block in Module 5.
Achievement
Complete all 5 lessons to finish the Multi-Head Attention module.
