https://datatweets.com/courses/transformers-from-scratch/training-a-tiny-gpt/Recent content in Training a Tiny GPT on DATATWEETSHugoenCopyright (c) 2026 DatatweetsFri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/training-a-tiny-gpt/lesson-1-batching-and-data-loading/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/training-a-tiny-gpt/lesson-1-batching-and-data-loading/Before you can train the mini-GPT you need a way to feed it data. This lesson builds the batching pipeline from scratch in NumPy: sampling random contiguous chunks of the corpus, shifting each chunk by one to make next-token targets, stacking them into a batch, and splitting off a validation set. Every shape and the one-token shift are verified on the real Lantern Bay corpus.https://datatweets.com/courses/transformers-from-scratch/training-a-tiny-gpt/lesson-2-the-adam-optimizer/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/training-a-tiny-gpt/lesson-2-the-adam-optimizer/Plain gradient descent uses one learning rate for every parameter, which stalls when different directions have wildly different curvature. This lesson builds Adam from scratch: first and second moment estimates, bias correction, and the per-parameter step. On an ill-conditioned test problem Adam drives the loss to zero while SGD, held to its largest stable rate, crawls — the reason Adam is the default for transformers.https://datatweets.com/courses/transformers-from-scratch/training-a-tiny-gpt/lesson-3-the-training-loop/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/training-a-tiny-gpt/lesson-3-the-training-loop/This is where the mini-GPT starts to learn. You’ll assemble the training loop — get a batch, forward to a loss, backward to gradients, Adam step, repeat — and run it end to end in NumPy on the Lantern Bay corpus. In about 24 seconds on a CPU the loss falls from the random baseline of 3.20 to around 0.13, the headline result of the whole course.https://datatweets.com/courses/transformers-from-scratch/training-a-tiny-gpt/lesson-4-monitoring-and-overfitting/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/training-a-tiny-gpt/lesson-4-monitoring-and-overfitting/Training loss alone can’t tell whether a model is learning or memorizing. This lesson adds the instrument that can: a held-out validation loss tracked alongside the training loss. On memorizable random data you’ll see the textbook overfitting signature — training loss falling while validation loss climbs above the random baseline — and you’ll watch a too-high learning rate make a run diverge.https://datatweets.com/courses/transformers-from-scratch/training-a-tiny-gpt/lesson-5-guided-project-train-your-mini-gpt/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/training-a-tiny-gpt/lesson-5-guided-project-train-your-mini-gpt/The module comes together into one real training run. You’ll configure a 154,456-parameter mini-GPT, train it in about 27 seconds of pure NumPy while tracking train and validation loss down to about 0.12, do a greedy continuation that produces a coherent Lantern Bay sentence, and save the trained weights to disk and reload them — the finished, trained model that Module 8 learns to sample from.