Lesson 4 - Monitoring & Overfitting

Welcome to Monitoring & Overfitting#

In the last lesson the training loss fell from 3.20 to 0.13 and it was tempting to call that success. But a low training loss only tells you the model fits the data it trained on — it says nothing about whether it has learned anything that generalizes. A model with enough capacity can drive its training loss to zero by simply memorizing, and the training loss will look wonderful the whole time. This lesson gives you the instrument that catches that: a validation loss, tracked alongside the training loss.

By the end of this lesson, you will be able to:

• Estimate training and validation loss during a run and say what each measures
• Recognize the overfitting signature: training loss down, validation loss up
• Explain why a validation loss above the random baseline means memorization
• Diagnose a learning rate that is set too high from the loss trajectory

Measuring Both Losses#

The setup from Lesson 1 already holds out a validation split. To monitor a run, we periodically estimate the loss on both sets — averaging over a few batches for a stable number rather than trusting a single noisy batch. Training loss measures how well the model fits what it learns from; validation loss measures how well that fit transfers to text it never trained on. When the two move together, the model is genuinely learning. When they diverge, it is memorizing.

def estimate_loss(model, data, block_size, batch_size, rng, iters=20):
    total = 0.0
    for _ in range(iters):
        xb, yb = get_batch(data, block_size, batch_size, rng)
        _, loss = model.forward(xb, yb)
        total += loss
    return total / iters

On the real Lantern Bay corpus, training and validation loss stay locked together (both land near 0.13), because the corpus is so repetitive that anything the model learns from one part transfers directly to another — there is nothing to overfit to. That is the healthy case. To actually see overfitting, we need data with no generalizable structure at all.

The Overfitting Signature#

The cleanest demonstration of overfitting is to train on random tokens. Random data has no pattern to learn, so the only way to lower the training loss is to memorize the specific training sequence — and memorizing the training set does nothing for a fresh random validation set. So the training loss falls while the validation loss, having nothing to gain, gets worse.

import numpy as np

rng = np.random.default_rng(0)
train_rand = rng.integers(0, vocab_size, 600)   # structureless: only memorizable
val_rand = rng.integers(0, vocab_size, 600)     # a different random draw

model = MiniGPT(vocab_size, block_size=16, n_embd=64, n_head=4, n_layer=2, seed=42)
opt = Adam(model.P, lr=3e-3)
r = np.random.default_rng(1)

print(f"random baseline log(24) = {np.log(24):.3f}")
for step in range(801):
    xb, yb = get_batch(train_rand, 16, 16, r)
    _, loss = model.forward(xb, yb)
    grads = model.backward()
    opt.step(model.P, grads)
    if step % 100 == 0:
        vl = estimate_loss(model, val_rand, 16, 16, r)
        print(f"step {step:3d}  train {loss:.4f}  val {vl:.4f}  gap {vl - loss:+.4f}")

random baseline log(24) = 3.178
step   0  train 3.1923  val 3.2564  gap +0.0641
step 100  train 1.9414  val 4.2739  gap +2.3325
step 200  train 0.7883  val 5.6863  gap +4.8981
step 300  train 0.4518  val 6.9382  gap +6.4863
step 400  train 0.3802  val 7.3688  gap +6.9886
step 500  train 0.4134  val 7.9525  gap +7.5391
step 600  train 0.3846  val 7.5570  gap +7.1724
step 700  train 0.3332  val 8.1373  gap +7.8041
step 800  train 0.2715  val 8.6100  gap +8.3385

This is overfitting in its purest form. The training loss falls from 3.19 to 0.27 as the model memorizes the random training sequence, while the validation loss climbs from 3.26 all the way to 8.61 — and the gap widens monotonically to over +8. The most damning detail: the validation loss rises above the random baseline of 3.18. The model is now worse than random guessing on unseen data, because it has become confidently wrong — assigning high probability to whatever it memorized, which is useless for the fresh sequence. If you only watched the training loss, you would think this run was going beautifully.

When the Learning Rate Is Too High#

The other thing to watch for is a learning rate that is too large. Adam is forgiving, but past a point each step overshoots so badly that the loss increases, sometimes exploding toward infinity or NaN. The signature is unmistakable: the loss goes up.

rng = np.random.default_rng(1)
model = MiniGPT(vocab_size, block_size=16, n_embd=32, n_head=4, n_layer=2, seed=1)
opt = Adam(model.P, lr=0.5)          # far too high
for step in range(6):
    xb, yb = get_batch(data, 16, 16, rng)
    _, loss = model.forward(xb, yb)
    grads = model.backward()
    opt.step(model.P, grads)
    print(f"lr=0.5  step {step}  loss {loss:.4f}")

lr=0.5  step 0  loss 3.1638
lr=0.5  step 1  loss 6.8110
lr=0.5  step 2  loss 15.5328
lr=0.5  step 3  loss 12.2288
lr=0.5  step 4  loss 11.8275
lr=0.5  step 5  loss 13.6507

At lr=0.5 the loss more than doubles on the very first step and lurches around above 10 — the optimizer is stepping so far it overshoots the minimum every time. The fix is simply a smaller learning rate: the 3e-3 we used in Lesson 3 keeps every step in the regime where the loss actually goes down. When a run’s loss climbs from step one, the learning rate is almost always the first thing to lower.

Practice Exercises#

Exercise 1: The gap over time#

From the overfitting run, plot or list val - train at each logged step. Is the gap widening, narrowing, or stable, and what does that tell you about when you should have stopped training?

Exercise 2: Real corpus, both losses#

Run the Lesson 3 training loop but also call estimate_loss on val_data every 100 steps. Do train and validation track together? What does that say about whether the model is overfitting the Lantern Bay text?

Exercise 3: Find the stable learning rate#

Try lr = 0.1, 0.03, 0.01 on the real corpus for 50 steps each. Which ones make the loss fall smoothly, and which make it spike or stall?

Summary#

A falling training loss is necessary but not sufficient — only a held-out validation loss reveals whether the model is learning or memorizing. On structureless random data the overfitting signature is stark: training loss falls to 0.27 while validation loss climbs to 8.61, above the random baseline, because the model has become confidently wrong on unseen data. And a learning rate set too high (0.5) makes the loss climb from the first step. Monitoring both losses, and watching for a rising loss, is how you keep a training run honest.

Key Concepts#

• Validation loss: loss on held-out data; measures generalization, not fit.
• Overfitting signature: training loss down while validation loss flattens or rises.
• Above-baseline validation: a validation loss exceeding log(vocab_size) means confidently-wrong memorization.
• Diverging loss: a loss that rises from step one signals a learning rate set too high.

Why This Matters#

Every serious training run is judged on validation, not training, loss — because the entire point of a model is to work on data it hasn’t seen. The overfitting signature you just produced deliberately is the same one that decides when to stop training a real language model, how big a model the data can support, and whether a change helped or hurt. And recognizing a diverging loss instantly, and reaching for the learning rate, is the single most common piece of training first aid. These two habits — watch validation, watch for divergence — are what separate a run you can trust from a number you happen to like.

Continue Building Your Skills#

You can now train a model and tell whether the training is honest. The next lesson brings the whole module together into one guided project: you’ll configure a run, train the mini-GPT end to end while monitoring both losses, watch it generate recognizable Lantern Bay text, and save the trained weights to disk — producing the finished, trained model that Module 8 will learn to sample from in interesting ways.