Title here
Summary here
Module · 5 lessons
Training a Tiny GPT
Your mini-GPT is built and gradient-checked — now train it. Learn batching and data loading, the Adam optimizer, the training loop that actually drops the loss, and how to monitor training and overfitting — then train your GPT on the Lantern Bay corpus in NumPy until it learns the text.
At a glance
Level
Advanced
Lessons
5 lessons
Time to complete
1 week
Cost
Free forever · no sign-up
Welcome to Training a Tiny GPT, the seventh module — the moment the model stops being a static piece of math and starts learning. You’ve built the mini-GPT and proven every gradient correct. Now you’ll feed it data, run the optimization, and watch the loss fall from the random baseline of about 3.18 toward a model that has genuinely absorbed the Lantern Bay text.
You’ll build each piece of the training machinery from scratch. Batching and data loading samples random fixed-length chunks from the corpus and shifts them into input/target pairs. The Adam optimizer — with its running averages of the gradient and its square, plus bias correction — is what makes transformer training converge quickly, and you’ll implement it yourself. Then the training loop ties it together: forward, loss, backward, update, repeat — and you’ll run it for real and see the loss drop. Finally you’ll learn to monitor training, splitting train from validation to watch overfitting on the small corpus and reading the loss curves the way a practitioner does. The guided project runs a complete training session and hands you a trained model, ready to generate text in Module 8.
Start with Lesson 1, where the corpus becomes a stream of training batches.
Lessons in this module
1
Batching & Data Loading
Turn a long token stream into shuffled training batches: sample random fixed-length chunks, shift them into input and target pairs, and hold out a validation split — the data pipeline that feeds GPT training.
2
The Adam Optimizer
Implement the Adam optimizer from scratch in NumPy — running averages of the gradient and its square, plus bias correction — and see why its per-parameter adaptive steps train transformers where plain gradient descent stalls.
3
The Training Loop
Connect the batches, the model, and the optimizer into a training loop, and run it for real: forward for the loss, backward for the gradients, an Adam step, repeat — and watch the mini-GPT's loss fall from the random baseline toward a model that has learned the text.
4
Monitoring & Overfitting
A falling training loss can lie. Track training and validation loss together to tell learning from memorizing, watch the unmistakable overfitting signature on data with no structure to generalize, and see what a learning rate set too high does to a run.
5
Guided Project: Train Your Mini-GPT
Run a complete training session end to end: configure the model, train it on the Lantern Bay corpus while monitoring train and validation loss, watch it generate recognizable text, and save the trained weights — producing the finished model you'll sample from next.
Achievement
Complete all 5 lessons to finish the Training a Tiny GPT module.
