Title here
Summary here
Course
Transformers from Scratch
Build a transformer from nothing but NumPy and the chain rule — scaled dot-product attention, multi-head attention, positional encodings, the transformer block, a causal mask, and a tiny GPT you train and sample from, with every matrix multiply and gradient written by hand
At a glance
Level
Advanced
Lessons
41 lessons across 9 modules
What you build
A working mini-GPT, from scratch in NumPy
Cost
100% free · no GPU, no API key · NumPy only
What you'll build
You'll build a transformer from scratch — no PyTorch, no TensorFlow, no Hugging Face — using nothing but NumPy and the chain rule. Starting from the gap that recurrent networks leave behind, you'll implement scaled dot-product attention, stack it into multi-head attention, add positional encodings and embeddings, assemble the full transformer block with residual connections and layer norm, apply a causal mask to make it a GPT, and then train a tiny character-level GPT with a hand-written backward pass until the loss visibly falls and it generates text. Every forward shape is printed and checked, every gradient is verified against a numerical estimate, and the final model trains on an original corpus on your laptop's CPU in minutes.
Course syllabus
Work through the modules at your own pace. Each lesson is a self-contained, hands-on read.
1 From Sequences to Attention
2 Self-Attention from Scratch
3 Multi-Head Attention
4 Positional Encoding & Embeddings
5 The Transformer Block
6 Causal Masking & GPT
7 Training a Tiny GPT
8 Generation & Sampling
Before you start
This is an advanced course. You'll need comfortable NumPy and a working understanding of neural-network fundamentals from scratch — the forward pass, backpropagation and the chain rule, gradient descent, and softmax cross-entropy. Our
Deep Learning Foundations module (which builds a NumPy network and its backward pass by hand) and the
Sequence Models module are ideal preparation. If you've only ever used attention through a library, this course builds it from the ground up so you understand every matrix multiply.
Set up your environment
You can complete this entire course on any laptop with Python 3.10+ and NumPy — no GPU, no deep-learning framework, and no API key. That's the point: when you write every operation yourself, there's nowhere for the magic to hide.
- Install the one package the course uses:
pip install numpyEvery array shape, every attention weight, and every training loss in this course is produced by real NumPy code with a fixed random seed, so your numbers will match the ones shown and you can rerun any experiment end to end.
We use a deliberately tiny model and a small original corpus so the from-scratch training loop runs in minutes on a CPU. The architecture is exactly the one behind modern large language models — only the scale is different.
Ready to build a GPT from nothing but NumPy?
Start with the limits of recurrence — the problem attention was invented to solve — and work through every matrix multiply from a single attention head to a tiny GPT that writes text.
Start the first lessonWant this taught live to your team?
Mehdi runs tailored corporate workshops on this exact material — hands-on, in-person or remote.
