Title here
Summary here
Module · 5 lessons
Causal Masking & GPT
Turn the transformer block into a GPT: the autoregressive next-token objective, the causal mask that stops a position from seeing the future, the decoder-only stack, and the language-modeling head and loss — assembling a complete mini-GPT in NumPy, forward and backward.
At a glance
Level
Advanced
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Causal Masking & GPT, the sixth module — where all the parts click together into a real language model. You have attention, multi-head attention, positional embeddings, and the transformer block. The one thing standing between that and a GPT is a rule about time: to predict the next token, a position must never be allowed to peek at the tokens that come after it.
You’ll start with the autoregressive idea — that language modeling is just next-token prediction, done at every position in parallel. Then you’ll implement the causal mask that enforces it, setting the attention scores above the diagonal to negative infinity so each position attends only to itself and the past. You’ll stack causal transformer blocks into the decoder-only architecture that GPT uses, and add the language-modeling head — a projection to vocabulary logits — with the softmax cross-entropy next-token loss whose gradient you’ll derive and check. The guided project assembles the whole thing into a mini-GPT: token and position embeddings, a stack of causal blocks, a final layer norm, and the output head — a complete model, forward and backward, ready for the training loop in Module 7.
Start with Lesson 1, where predicting the next character becomes the only objective that matters.
Lessons in this module
1
The Autoregressive Idea
Frame language modeling as autoregressive next-token prediction, build shifted (input, target) pairs in NumPy, and compute the uniform-random loss baseline your GPT must beat.
2
The Causal Mask
Implement the causal mask that makes self-attention autoregressive: set future scores to negative infinity before softmax so each position attends only to itself and the past, then verify the backward pass zeros the masked gradients.
3
The Decoder-Only Stack
Assemble a GPT forward pass in NumPy: token and position embeddings, a stack of causal transformer blocks, a final layer norm, and an output head that produces next-token logits.
4
The Language-Modeling Head & Loss
Project the transformer's hidden states to vocabulary logits with the LM head, then turn them into a trainable softmax cross-entropy next-token loss in pure NumPy, gradient-checked.
5
Guided Project: Assembling the Full GPT
Package token and position embeddings, a stack of causal transformer blocks, a final LayerNorm, and the language-modeling head into one MiniGPT NumPy class — forward, backward, and gradient-checked.
Achievement
Complete all 5 lessons to finish the Causal Masking & GPT module.
