https://datatweets.com/courses/transformers-from-scratch/the-transformer-block/Recent content in The Transformer Block on DATATWEETSHugoenCopyright (c) 2026 DatatweetsFri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/the-transformer-block/lesson-1-residual-connections/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/the-transformer-block/lesson-1-residual-connections/Residual connections compute out = x + sublayer(x). This lesson shows why that trivial-looking addition is the reason deep transformers train at all: because the derivative of the + x path is 1, every layer gets a direct gradient highway to the loss. You build a real NumPy demo that stacks up to 20 attenuating sublayers and measures the gradient norm reaching the input with and without residuals, watch the plain stack collapse to 8e-07 while the residual stack stays healthy, and gradient-check the backward pass in float64.https://datatweets.com/courses/transformers-from-scratch/the-transformer-block/lesson-2-layer-normalization/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/the-transformer-block/lesson-2-layer-normalization/Layer normalization standardizes each token’s feature vector on its own, across the feature dimension, so a deep stack of transformer blocks stays trainable regardless of batch or sequence length. This lesson builds layernorm_forward and layernorm_backward in float64 NumPy, derives the subtle dx formula that accounts for the mean and variance both depending on x, and gradient-checks all three gradients to a max relative error near 1e-8.https://datatweets.com/courses/transformers-from-scratch/the-transformer-block/lesson-3-the-feed-forward-network/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/the-transformer-block/lesson-3-the-feed-forward-network/Attention mixes information across positions; the feed-forward network gives each position room to think. In this lesson you build the position-wise FFN from scratch in NumPy: two linear layers with a ReLU between them that expand the width from C to 4C and project it back to C, applied independently to every position. You implement ffn_forward with caching and ffn_backward (dW2, db2, the ReLU, dW1, db1, dx), then prove the backward pass with a float64 numerical gradient check whose max relative errors land near 1e-7.https://datatweets.com/courses/transformers-from-scratch/the-transformer-block/lesson-4-assembling-the-block/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/the-transformer-block/lesson-4-assembling-the-block/This lesson combines the module’s pieces into one repeatable unit: the pre-norm transformer block. You wrap multi-head attention and the feed-forward network each in a residual connection, normalize on the way in with LayerNorm, and confirm the block maps (B, T, C) to (B, T, C). You implement transformer_block_forward for a seed-42 demo (B=1, T=6, C=64, h=4), print the shape at every sub-step, then stack two and then six blocks to prove the shape-preserving property is exactly what makes deep transformers possible.https://datatweets.com/courses/transformers-from-scratch/the-transformer-block/lesson-5-guided-project-transformer-block-class/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/the-transformer-block/lesson-5-guided-project-transformer-block-class/In this guided project you fold the whole module into a single reusable NumPy class. You build TransformerBlock(C, h) holding LayerNorm1 (gamma1, beta1), multi-head attention (Wq, Wk, Wv, Wo), LayerNorm2 (gamma2, beta2), and a feed-forward network (W1, b1, W2, b2, hidden 4C), add a pre-norm forward that caches everything, and a backward that walks the two residual sublayers in reverse to fill every parameter gradient and return dx. You prove it with a float64 finite-difference gradient check whose worst relative error across all twelve parameters and the input is 6.6e-07, then watch an MSE loss fall monotonically from 1.5964 to 1.1200. Everything is seeded and byte-for-byte reproducible.