https://datatweets.com/courses/transformers-from-scratch/causal-masking-and-gpt/Recent content in Causal Masking & GPT on DATATWEETSHugoenCopyright (c) 2026 DatatweetsFri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/causal-masking-and-gpt/lesson-1-the-autoregressive-idea/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/causal-masking-and-gpt/lesson-1-the-autoregressive-idea/A language model factorizes the probability of a sequence into a product of next-token conditionals, and training just maximizes the probability it assigns to the true next character at every position. This lesson makes that concrete: you tokenize a slice of the Lantern Bay corpus, build (input, target) pairs by shifting the tokens by one so targets[t] equals inputs[t+1], and see why a transformer can score all T next-token predictions in a single parallel forward pass where an RNN would need T steps. You finish by computing the uniform-random baseline loss, log(24) = 3.178 nats, the number training has to beat.https://datatweets.com/courses/transformers-from-scratch/causal-masking-and-gpt/lesson-2-the-causal-mask/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/causal-masking-and-gpt/lesson-2-the-causal-mask/Plain self-attention lets every position peek at every other, including future tokens — for next-token prediction that is cheating. This lesson adds the causal mask: set the attention scores above the diagonal to a large negative value before softmax so those weights collapse to zero and the attention matrix becomes lower-triangular. You build a real NumPy causal head (seed 42, float64), print the (T,T) matrix and confirm it is lower-triangular with rows summing to 1, show position 0 attends only to itself with weight 1.0, and gradient-check the masked backward pass to a max relative error near 1e-9.https://datatweets.com/courses/transformers-from-scratch/causal-masking-and-gpt/lesson-3-the-decoder-only-stack/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/causal-masking-and-gpt/lesson-3-the-decoder-only-stack/This lesson wires every piece you have built into one model: the decoder-only GPT. You embed token ids and add positions, run the result through a stack of n_layer causal transformer blocks, apply a final layer norm, and project to vocabulary logits with an output head. You implement gpt_forward in pure NumPy, print the shape at every stage, and confirm the forward pass turns a (1, 8) batch of token ids into (1, 8, 24) logits, one valid next-token distribution per position.https://datatweets.com/courses/transformers-from-scratch/causal-masking-and-gpt/lesson-4-the-language-modeling-head-and-loss/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/causal-masking-and-gpt/lesson-4-the-language-modeling-head-and-loss/The last piece before a real GPT: how hidden states become a loss you can train. You add the language-modeling head, a linear projection from (B,T,C) hidden states to (B,T,vocab_size) logits, then derive the softmax cross-entropy next-token loss whose target is the input shifted by one. You implement cross_entropy_loss returning both the loss and its famously clean gradient (softmax minus onehot, divided by N), verify it against a float64 numerical gradient check, confirm a random model’s loss lands at the uniform baseline log(vocab_size) around 3.178, and gradient-check dW_head and db_head.https://datatweets.com/courses/transformers-from-scratch/causal-masking-and-gpt/lesson-5-guided-project-assembling-the-full-gpt/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/causal-masking-and-gpt/lesson-5-guided-project-assembling-the-full-gpt/In this guided project you assemble everything the course has built into a single MiniGPT NumPy class: a token embedding E and position table P, n_layer causal transformer blocks, a final LayerNorm, and an LM head. You implement forward(idx, targets) that returns (B,T,24) logits and the next-token cross-entropy loss (starting near log 24 = 3.178), then backward() that flows the loss to every parameter using dQuery/dKey/dValue inside attention. You prove the whole model with a float64 gradient check whose worst relative error is 5.3e-05 across a slice of E, P, a block’s Wq and gamma1, and W_head, then overfit one fixed sequence with SGD and watch the loss plunge from 3.1788 to 0.0017. Everything is seeded and byte-for-byte reproducible.