https://datatweets.com/courses/transformers-from-scratch/multi-head-attention/Recent content in Multi-Head Attention on DATATWEETSHugoenCopyright (c) 2026 DatatweetsFri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/multi-head-attention/lesson-1-why-one-head-isnt-enough/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/multi-head-attention/lesson-1-why-one-head-isnt-enough/A single attention head, fixed by its one set of Wq, Wk, and Wv, produces exactly one attention distribution per query position, so it can only look for one kind of relationship at a time. This lesson motivates multi-head attention by building two independent NumPy heads over the same Lantern Bay sentence and showing their attention matrices are measurably different: a max absolute weight gap of 0.26 and disagreement on 7 of 8 rows about which word to focus on.https://datatweets.com/courses/transformers-from-scratch/multi-head-attention/lesson-2-splitting-into-heads/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/multi-head-attention/lesson-2-splitting-into-heads/One clean matmul with a wide (C, C) weight, then a reshape and a transpose, is all it takes to turn a (B, T, C) tensor into h independent (T, hs) attention problems. In this lesson you project a real (1, 6, 64) input, reshape it to (1, 6, 4, 16), transpose to (1, 4, 6, 16), and prove in NumPy that each head equals a separate small projection and that the split is perfectly invertible.https://datatweets.com/courses/transformers-from-scratch/multi-head-attention/lesson-3-parallel-attention-and-concatenation/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/multi-head-attention/lesson-3-parallel-attention-and-concatenation/This lesson completes the multi-head attention forward pass. Building on the split from Lesson 2, you run scaled dot-product attention in parallel across all heads with a single set of 4-D matmuls (scores (B,h,T,T), softmax over the last axis, per-head output (B,h,T,hs)), concatenate the heads back to (B,T,C), and mix them with an output projection Wo. You implement multi_head_attention_forward, print the shape at every stage for a seed-42 demo (B=2, T=6, C=64, h=4, hs=16), confirm each head’s attention rows sum to 1, and prove why Wo is needed by watching a block-diagonal Wo keep the heads siloed while a full Wo lets all four interact.https://datatweets.com/courses/transformers-from-scratch/multi-head-attention/lesson-4-the-multi-head-backward-pass/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/multi-head-attention/lesson-4-the-multi-head-backward-pass/This is the hardest lesson in the module: the backward pass through the complete multi-head attention forward from Lesson 3. You cache the forward, then walk the gradient back in order — dWo and dconcat through the output projection, split dconcat into per-head dheadout, run the batched single-head attention backward across the head axis to get dValue, dscores, dQuery and dKey, merge those back to (B, T, C), and finish at dWq, dWk, dWv and dX. Every gradient is verified against a float64 finite-difference estimate, with real printed max relative errors between 1e-7 and 1e-10.https://datatweets.com/courses/transformers-from-scratch/multi-head-attention/lesson-5-guided-project-multi-head-attention-layer/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/multi-head-attention/lesson-5-guided-project-multi-head-attention-layer/In this guided project you fold everything from Module 3 into a single reusable NumPy class. You build a MultiHeadAttention(C, h) skeleton holding Wq, Wk, Wv, and an output projection Wo, add a forward that projects, splits into (B, h, T, hs), runs scaled dot-product attention per head, concatenates, and mixes with Wo, then add a backward that returns dX and stores dWq/dWk/dWv/dWo. You prove it with a float64 finite-difference gradient check (max relative errors around 1e-7 or smaller) and a ten-step gradient-descent run that drives an MSE loss from 0.6508 down to 0.3840. Everything is seeded and byte-for-byte reproducible.