https://datatweets.com/courses/transformers-from-scratch/self-attention-from-scratch/Recent content in Self-Attention from Scratch on DATATWEETSHugoenCopyright (c) 2026 DatatweetsFri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/self-attention-from-scratch/lesson-1-q-k-v-projections/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/self-attention-from-scratch/lesson-1-q-k-v-projections/Self-attention lets a sequence attend to itself by projecting one input X through three separate learned matrices — Wq, Wk, and Wv — into a query space, a key space, and a value space. This lesson gives each projection its precise role, shape, and parameter count, then proves in pure NumPy that because Wq and Wk differ, Q and K differ and the score matrix QK-transpose is not symmetric — attention has a direction. You embed a slice of the Lantern Bay corpus, run the full head forward, and confirm every attention row is a valid distribution.https://datatweets.com/courses/transformers-from-scratch/self-attention-from-scratch/lesson-2-scaled-dot-product-attention/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/self-attention-from-scratch/lesson-2-scaled-dot-product-attention/This lesson presents the canonical attention formula, softmax(QKᵀ/√d_k)·V, one piece at a time, then proves why the scale matters. You run a real NumPy experiment (seed 42) showing that raw dot-product score variance grows with d_k — 3.99, 15.89, 64.25, 256.39 for d_k = 4, 16, 64, 256 — while dividing by √d_k pins it near 1.0 at every dimension. You watch an unscaled softmax at d_k = 256 collapse into a near one-hot spike (entropy 0.0003) while the scaled version stays spread out (entropy 1.79), then implement the reusable scaled_dot_product_attention forward and confirm its shapes and row sums.https://datatweets.com/courses/transformers-from-scratch/self-attention-from-scratch/lesson-3-the-attention-matrix/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/self-attention-from-scratch/lesson-3-the-attention-matrix/Learn to read the attention matrix your self-attention layer produces. You will take a real 10-character slice of the Lantern Bay corpus, embed it, run scaled dot-product attention in NumPy with seed 42, and print the (T, T) weight matrix. Then you will interpret focused versus diffuse rows, confirm every row sums to 1, prove the matrix is not symmetric because Wq and Wk differ, and see why identical characters produce identical rows until positional encoding arrives in Module 4.https://datatweets.com/courses/transformers-from-scratch/self-attention-from-scratch/lesson-4-the-backward-pass-for-attention/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/self-attention-from-scratch/lesson-4-the-backward-pass-for-attention/This is the hardest and most important lesson in the module: the backward pass through scaled dot-product self-attention. You cache the forward pass, then derive each gradient in order — dA and dValue through the value multiply, dS through the softmax Jacobian, dS_scaled through the scale, dQuery and dKey through the score matrix, and dWq, dWk, dWv, and dX back to the parameters and input. Every gradient is verified against a finite-difference numerical estimate in float64, with real printed max relative errors on the order of 1e-7 or smaller.https://datatweets.com/courses/transformers-from-scratch/self-attention-from-scratch/lesson-5-guided-project-self-attention-layer-class/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/self-attention-from-scratch/lesson-5-guided-project-self-attention-layer-class/In this guided project you turn everything from Module 2 into a single reusable NumPy class. You build a SelfAttention skeleton that holds Wq, Wk, Wv, add a forward method that caches (X, Q, K, V, A), add a backward method that returns dX and stores dWq/dWk/dWv, then prove it: a float64 finite-difference gradient check gives max relative errors around 1e-6 or smaller, and a short manual gradient-descent run drives a real MSE loss from 0.6470 down to 0.4835. Everything is seeded and byte-for-byte reproducible.