Title here
Summary here
Module · 5 lessons
The Transformer Block
Attention alone doesn't make a transformer. Learn the three pieces that make deep stacks trainable — residual connections, layer normalization, and the position-wise feed-forward network — then assemble them with multi-head attention into the repeatable transformer block, forward and backward, in NumPy.
At a glance
Level
Advanced
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to The Transformer Block, the fifth module. You’ve built attention; now you’ll build everything around it that turns a single clever operation into a deep, trainable network. A transformer is just this block, repeated — so getting it right is what makes the whole model work.
You’ll add the three ingredients that make depth possible. Residual connections give gradients a direct path backward, so a twenty-layer stack trains as easily as a two-layer one. Layer normalization keeps each position’s activations well-scaled from block to block — and you’ll derive its famously subtle backward pass and gradient-check it. The feed-forward network, a small position-wise MLP that expands and contracts the representation, gives each position room to compute. Then you’ll assemble all of it — multi-head attention from Module 3, layer norm, the feed-forward network, and residual connections — into the pre-norm transformer block, verify it end to end, and confirm it preserves shape so you can stack it as many times as you like. The guided project packages it into a clean, gradient-checked block class.
Start with Lesson 1, where a simple skip connection rescues gradients from the depths.
Lessons in this module
1
Residual Connections
Learn residual (skip) connections from scratch in NumPy and see why the added identity path keeps gradients alive in deep transformer stacks.
2
Layer Normalization
Learn layer normalization from scratch in NumPy and derive its subtle backward pass, then gradient-check dx, dgamma, and dbeta against a numerical gradient.
3
The Feed-Forward Network
Build the position-wise feed-forward network inside each transformer block from scratch in NumPy: expand to 4C, apply a nonlinearity, project back to C, then gradient-check the backward pass.
4
Assembling the Block
Assemble multi-head attention, two layer norms, the feed-forward network, and two residual adds into the pre-norm transformer block, then stack it in NumPy.
5
Guided Project: A Transformer Block Class
Package residual connections, layer normalization, multi-head attention, and the feed-forward network into one reusable, gradient-checked NumPy TransformerBlock class that stacks n_layer times to build the GPT.
Achievement
Complete all 5 lessons to finish the The Transformer Block module.
