Lesson 5 - Guided Project: Assembling the Full GPT

Welcome to the Guided Project#

This is the lesson the whole course has been building toward. You have token and position embeddings from Module 4, multi-head attention from Module 3, a gradient-checked transformer block from Module 5, and in this module you added the one rule that turns a transformer into a language model: the causal mask that forbids a position from attending to the future. Every piece is verified. What is missing is the object that holds them all and runs them as one model.

In this guided project you build that object: a single MiniGPT class with a forward and a backward method. It owns a token embedding E, a position table P, a list of n_layer causal transformer blocks, a final LayerNorm, and the language-modeling head (W_head, b_head) that projects each position to a distribution over the 24 characters in our vocabulary. forward(idx, targets) embeds the tokens, runs them through the causal stack, normalizes, and produces (B, T, 24) logits plus the next-token cross-entropy loss. backward() sends that single scalar loss all the way back to every parameter — through the head, the final norm, each causal block in reverse, and finally into the embeddings.

We use a deliberately small config — n_embd=32, n_head=4, n_layer=2, block_size=16 — so the whole-model gradient check runs in seconds. And we finish the way every serious component in this course finishes: a float64 gradient check that proves backward computes the true derivative of forward, and a tiny overfitting run that proves the assembled model can actually learn. This exact MiniGPT is what you train for real on the full corpus in Module 7, so we keep it clean and self-contained.

By the end of this project, you will be able to:

• Structure a complete GPT as one class owning embeddings, causal blocks, a final norm, and an LM head plus their gradients
• Implement forward(idx, targets) that embeds tokens, runs the causal stack, and returns logits and the next-token cross-entropy loss
• Implement backward() that flows the loss through the head, final norm, every block in reverse, and back into E and P using dQuery/dKey/dValue inside attention
• Gradient-check the entire model in float64 and confirm the worst relative error stays under 1e-4
• Overfit a single fixed sequence with gradient descent and watch the loss fall far below the log(24) baseline

This project assumes the earlier lessons in this module (the autoregressive objective, the causal mask, the decoder-only stack, and the next-token loss) and the TransformerBlock class from Module 5. You only need numpy — no PyTorch, no GPU, no API key.

Stage 1: The Class#

A GPT is a stack of parts, and __init__ is where we lay them out. The model owns exactly five kinds of parameter: the token embedding E of shape (vocab_size, C), one row per character; the position table P of shape (block_size, C), one row per slot in the context window; a list of n_layer causal transformer blocks; the final LayerNorm (gamma_f, beta_f); and the LM head W_head of shape (C, vocab_size) with bias b_head, which turns each position’s C-vector into 24 logits. Everything is seeded through one default_rng(42) and drawn small at * 0.02, the standard transformer init.

The blocks are the same TransformerBlock you gradient-checked in Module 5, with one change made earlier in this module: its attention applies the causal mask before the softmax, so position t attends only to positions t and earlier. We include the class here so the project runs on its own, but nothing about it is new — it is your verified block with the mask baked in.

import numpy as np
import warnings
warnings.filterwarnings("ignore")

def softmax(x, axis=-1):
    x = x - x.max(axis=axis, keepdims=True)      # stable: subtract row max
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)

class TransformerBlock:
    """Pre-norm CAUSAL transformer block (Module 5 block + the causal mask from L2)."""

    def __init__(self, C, h, rng, eps=1e-5):
        assert C % h == 0
        self.C, self.h, self.hs, self.eps = C, h, C // h, eps
        self.gamma1 = np.ones(C); self.beta1 = np.zeros(C)
        self.Wq = rng.standard_normal((C, C)) * 0.02
        self.Wk = rng.standard_normal((C, C)) * 0.02
        self.Wv = rng.standard_normal((C, C)) * 0.02
        self.Wo = rng.standard_normal((C, C)) * 0.02
        self.gamma2 = np.ones(C); self.beta2 = np.zeros(C)
        self.W1 = rng.standard_normal((C, 4 * C)) * 0.02
        self.b1 = np.zeros(4 * C)
        self.W2 = rng.standard_normal((4 * C, C)) * 0.02
        self.b2 = np.zeros(C)
        self.cache = None

    def _ln_forward(self, x, gamma, beta):
        mu = x.mean(-1, keepdims=True)
        xc = x - mu
        var = (xc ** 2).mean(-1, keepdims=True)
        inv = 1.0 / np.sqrt(var + self.eps)
        xhat = xc * inv
        return gamma * xhat + beta, (xhat, inv, gamma)

    def _mha_forward(self, x):
        B, T, C = x.shape
        h, hs = self.h, self.hs
        Q, K, V = x @ self.Wq, x @ self.Wk, x @ self.Wv
        Qh = Q.reshape(B, T, h, hs).transpose(0, 2, 1, 3)
        Kh = K.reshape(B, T, h, hs).transpose(0, 2, 1, 3)
        Vh = V.reshape(B, T, h, hs).transpose(0, 2, 1, 3)
        scores = Qh @ Kh.transpose(0, 1, 3, 2) / np.sqrt(hs)
        mask = np.triu(np.ones((T, T), dtype=bool), k=1)      # causal: block the future
        scores = np.where(mask, -1e9, scores)
        A = softmax(scores, axis=-1)
        Oh = A @ Vh
        concat = Oh.transpose(0, 2, 1, 3).reshape(B, T, C)
        out = concat @ self.Wo
        return out, (x, Qh, Kh, Vh, A, concat, mask)

    def _ffn_forward(self, x):
        z = x @ self.W1 + self.b1
        hrelu = np.maximum(z, 0.0)
        return hrelu @ self.W2 + self.b2, (x, z, hrelu)

    def forward(self, x):
        ln1, ln1c = self._ln_forward(x, self.gamma1, self.beta1)
        mha_out, mhac = self._mha_forward(ln1)
        a = x + mha_out                                        # residual 1
        ln2, ln2c = self._ln_forward(a, self.gamma2, self.beta2)
        ffn_out, ffnc = self._ffn_forward(ln2)
        out = a + ffn_out                                      # residual 2
        self.cache = (ln1c, mhac, ln2c, ffnc)
        return out

class MiniGPT:
    """A complete decoder-only GPT: embeddings -> causal blocks -> final LN -> LM head."""

    def __init__(self, vocab_size, block_size, n_embd, n_head, n_layer, seed=42, eps=1e-5):
        self.vocab_size, self.block_size = vocab_size, block_size
        self.n_embd, self.n_head, self.n_layer, self.eps = n_embd, n_head, n_layer, eps
        rng = np.random.default_rng(seed)
        C = n_embd
        self.E = rng.standard_normal((vocab_size, C)) * 0.02   # token embedding
        self.P = rng.standard_normal((block_size, C)) * 0.02   # position table
        self.blocks = [TransformerBlock(C, n_head, rng, eps) for _ in range(n_layer)]
        self.gamma_f = np.ones(C); self.beta_f = np.zeros(C)   # final LayerNorm
        self.W_head = rng.standard_normal((C, vocab_size)) * 0.02
        self.b_head = np.zeros(vocab_size)

vocab_size, block_size, n_embd, n_head, n_layer = 24, 16, 32, 4, 2
model = MiniGPT(vocab_size, block_size, n_embd, n_head, n_layer, seed=42)
print("E (vocab, C)   :", model.E.shape)
print("P (block, C)   :", model.P.shape)
print("n blocks       :", len(model.blocks), "| head size hs:", model.blocks[0].hs)
print("gamma_f/beta_f :", model.gamma_f.shape, model.beta_f.shape)
print("W_head/b_head  :", model.W_head.shape, model.b_head.shape)

E (vocab, C)   : (24, 32)
P (block, C)   : (16, 32)
n blocks       : 2 | head size hs: 8
gamma_f/beta_f : (32,) (32,)
W_head/b_head  : (32, 24)  (24,)

That is the entire model on paper: an embedding lookup at the bottom, two causal blocks in the middle, a normalization, and a projection to 24 logits at the top. Notice what __init__ commits to and what it leaves open. It fixes the widths — vocab_size, block_size, n_embd, n_head, n_layer — but never the batch size B or the actual sequence length T you feed in, exactly like the block did. The head size falls out as n_embd // n_head = 8. This small config has just under 27,300 parameters total, tiny enough that the pure-NumPy gradient check ahead finishes in seconds, yet it is structurally identical to a real GPT: shrink the numbers and you have a toy; grow them and you have GPT-2.

Stage 2: forward(idx, targets)#

forward runs the model top to bottom and, when given targets, reports how surprised it is by the true next character. The steps read straight off the architecture: look up E[idx] and add the positions P[:T]; push the result through every causal block in order; apply the final LayerNorm; and project to logits with the LM head. If targets is supplied, we also compute the next-token cross-entropy loss — softmax the logits over the vocabulary, and average the negative log-probability the model assigned to the correct next character. Along the way we cache the few values backward will need.

class MiniGPT:
    # ... __init__ from Stage 1 ...

    def _ln_forward(self, x, gamma, beta):
        mu = x.mean(-1, keepdims=True)
        xc = x - mu
        var = (xc ** 2).mean(-1, keepdims=True)
        inv = 1.0 / np.sqrt(var + self.eps)
        xhat = xc * inv
        return gamma * xhat + beta, (xhat, inv, gamma)

    def forward(self, idx, targets=None):
        B, T = idx.shape
        assert T <= self.block_size
        tok = self.E[idx]                       # (B, T, C) token embeddings
        pos = self.P[:T]                        # (T, C) position embeddings
        x = tok + pos                           # broadcast over the batch
        for blk in self.blocks:
            x = blk.forward(x)                  # causal transformer blocks
        xf, lnfc = self._ln_forward(x, self.gamma_f, self.beta_f)   # final LN
        logits = xf @ self.W_head + self.b_head                     # (B, T, vocab)
        if targets is None:
            self.cache = (idx, xf, lnfc)
            return logits, None
        probs = softmax(logits, axis=-1)
        N = B * T
        ll = np.log(probs.reshape(N, -1)[np.arange(N), targets.reshape(N)] + 1e-12)
        loss = -ll.mean()
        self.cache = (idx, xf, lnfc, probs, targets)
        return logits, loss

The loss deserves a closer look. We flatten the (B, T) grid of positions into N = B*T independent next-token predictions, pull out the probability each position gave to its true next character, and average the negative logs. Every position in every sequence is a training example, which is what makes language modeling so data-efficient: a length-16 sequence yields 16 predictions, not one. To see the model in action we draw a small batch from our Lantern Bay corpus — the same original passage the whole course uses — where each input row is 16 characters and each target row is that same window shifted one character to the right.

CORPUS = (
    "the lamp keeper walks the shore at dusk. "
    "the lamp keeper lights the lantern when the fog rolls in. "
    "the boats come home when the lantern glows. "
    "the boats wait outside the bay when the fog is thick. "
    "a small boat drifts near the rocks at dawn. "
    "the keeper rings the bell when a boat drifts near the rocks. "
    "the tide rises at dusk and falls at dawn. "
    "the gulls call over the bay when the tide is low. "
    "the keeper counts the boats and writes the count in a book. "
    "the lantern burns all night and rests at dawn. "
) * 20
chars = sorted(set(CORPUS))
stoi = {c: i for i, c in enumerate(chars)}
itos = {i: c for i, c in enumerate(chars)}
encode = lambda s: [stoi[c] for c in s]
decode = lambda ids: "".join(itos[i] for i in ids)
data = np.array(encode(CORPUS), dtype=np.int64)

rng = np.random.default_rng(42)
model = MiniGPT(vocab_size, block_size, n_embd, n_head, n_layer, seed=42)
B, T = 4, block_size
ix = rng.integers(0, len(data) - T - 1, size=B)
xb = np.stack([data[i:i + T] for i in ix])              # (4, 16) inputs
yb = np.stack([data[i + 1:i + 1 + T] for i in ix])      # (4, 16) next chars
logits, loss = model.forward(xb, yb)
print("idx shape :", xb.shape)
print("logits    :", logits.shape)
print("init loss :", round(float(loss), 4), "| log(24) =", round(float(np.log(24)), 4))

idx shape : (4, 16)
logits    : (4, 16, 24)
init loss : 3.2001 | log(24) = 3.1781

Two things confirm the forward pass is wired correctly. First, the logits come out (4, 16, 24): for every one of the 4 sequences and every one of the 16 positions, a full distribution over the 24 possible next characters. Second, the initial loss is 3.2001, right next to log(24) = 3.1781. That is the signature of a freshly initialized language model: with tiny random weights it has no idea what comes next, so it spreads its probability almost uniformly across all 24 characters, and a uniform guess over 24 options costs exactly log(24) nats. Just as a fresh binary classifier reports a loss near log(2), a fresh 24-way character model reports a loss near log(24) — a free sanity check that the embeddings, the causal stack, the final norm, the head, and the loss are all connected.

Stage 3: backward()#

Now we send that single scalar loss backward to every parameter in the model. The order is the exact reverse of forward: start at the loss, pass through the LM head, then the final LayerNorm, then each causal block in reverse (each block’s own backward handles its two residual sublayers, using dQuery, dKey, and dValue inside attention), and finally deposit gradients into the two embedding tables.

The head and loss combine into the cleanest step in all of backpropagation. Because softmax followed by cross-entropy collapses, the gradient of the loss with respect to the logits is just the predicted probabilities minus a one-hot of the truth, divided by the number of predictions:

The two embeddings need care of a different kind. E[idx] is a gather: many positions may look up the same character row, so its gradient is a scatter-add — every position sends its gradient back to whichever row it read, and rows read multiple times accumulate. The position table P is added to every sequence in the batch, so its gradient is the input gradient summed over the batch dimension.

class MiniGPT:
    # ... __init__, _ln_forward, forward ...

    def _ln_backward(self, dout, cache):
        xhat, inv, gamma = cache
        C = dout.shape[-1]
        dgamma = (dout * xhat).sum(axis=(0, 1))
        dbeta = dout.sum(axis=(0, 1))
        dxhat = dout * gamma
        dx = (inv / C) * (C * dxhat
                          - dxhat.sum(-1, keepdims=True)
                          - xhat * (dxhat * xhat).sum(-1, keepdims=True))
        return dx, dgamma, dbeta

    def backward(self):
        idx, xf, lnfc, probs, targets = self.cache
        B, T = idx.shape
        C, V, N = self.n_embd, self.vocab_size, B * T
        # 1. cross-entropy + softmax: dlogits = (probs - onehot) / N
        dlogits = probs.reshape(N, V).copy()
        dlogits[np.arange(N), targets.reshape(N)] -= 1.0
        dlogits /= N
        dlogits = dlogits.reshape(B, T, V)
        # 2. LM head
        self.dW_head = np.einsum('btc,btv->cv', xf, dlogits)
        self.db_head = dlogits.sum(axis=(0, 1))
        dxf = dlogits @ self.W_head.T
        # 3. final LayerNorm
        dx, self.dgamma_f, self.dbeta_f = self._ln_backward(dxf, lnfc)
        # 4. causal blocks, in reverse
        for blk in reversed(self.blocks):
            dx = blk.backward(dx)
        # 5. embeddings: x = E[idx] + P[:T]
        self.dE = np.zeros_like(self.E)
        np.add.at(self.dE, idx, dx)             # scatter-add token grads
        self.dP = np.zeros_like(self.P)
        self.dP[:T] = dx.sum(axis=0)            # position grads, summed over batch
        return dx

model.backward()
print("dE     :", model.dE.shape)
print("dP     :", model.dP.shape)
print("dW_head:", model.dW_head.shape, "| db_head:", model.db_head.shape)
print("block0 dWq:", model.blocks[0].dWq.shape, "| dgamma1:", model.blocks[0].dgamma1.shape)

dE     : (24, 32)
dP     : (16, 32)
dW_head: (32, 24) | db_head: (24,)
block0 dWq: (32, 32) | dgamma1: (32,)

Every gradient has the same shape as the parameter it belongs to — the first sanity check, and it passes cleanly from the head all the way down to the embeddings. The block gradients (dWq, dgamma1, and the nine others per block) are filled in by each block’s own backward, which we verified in Module 5; here they are simply driven by the dx that arrives from the final norm above. Notice how little the top-level backward has to do: five short steps, because all the hard work of attention and the feed-forward network is delegated to the blocks. This is the payoff of building verified components — assembling them is almost bookkeeping. Matching shapes, though, is necessary but not sufficient. The next stage proves the numbers are right.

Stage 4: Verify the Whole Model#

A backward pass you cannot verify is one you cannot trust, and this one threads a scalar loss through a head, a norm, two full transformer blocks, and two embedding tables. We prove it two ways: a gradient check (do the analytic gradients match finite differences across the whole model?) and an overfit test (does the assembled model actually learn?).

Gradient Check#

The idea is the definition of a derivative. Nudge one entry of one parameter by $\pm \epsilon$, measure how much the loss changes, and divide — that finite difference must match the analytic gradient:

We run everything in float64 — finite differences are numerically delicate, and single precision would drown the signal in rounding noise. The model has ~27,300 parameters, so instead of checking every one we sample a representative slice from each part of the model: a few rows of the token embedding E (only rows that actually appear in the batch have nonzero gradient), a slice of the position table P, one block’s query matrix Wq and its gamma1, and a slice of the LM head W_head. If the derivative is correct in a slice of every distinct component, it is correct everywhere.

def rel_err(a, b):
    return np.max(np.abs(a - b) / np.maximum(1e-12, np.abs(a) + np.abs(b)))

def loss_only(model, xb, yb):
    return model.forward(xb, yb)[1]

def numeric_grad_subset(model, xb, yb, P, idxs, eps=1e-6):
    g = np.zeros(len(idxs))
    for k, i in enumerate(idxs):
        old = P[i]
        P[i] = old + eps; lp = loss_only(model, xb, yb)
        P[i] = old - eps; lm = loss_only(model, xb, yb)
        P[i] = old
        g[k] = (lp - lm) / (2 * eps)
    return g

rngc = np.random.default_rng(0)
model = MiniGPT(vocab_size, block_size, n_embd, n_head, n_layer, seed=42)
Bc, Tc = 2, 8                                    # small batch keeps the check fast
ixc = rngc.integers(0, len(data) - Tc - 1, size=Bc)
xc = np.stack([data[i:i + Tc] for i in ixc])
yc = np.stack([data[i + 1:i + 1 + Tc] for i in ixc])
model.forward(xc, yc)
model.backward()

used = sorted(set(xc.reshape(-1).tolist()))[:4]  # rows of E that appear in the batch
checks = [
    ("E",         model.E,                model.dE,                [(t, j) for t in used for j in range(4)]),
    ("P",         model.P,                model.dP,                [(t, j) for t in range(4) for j in range(4)]),
    ("block0.Wq", model.blocks[0].Wq,     model.blocks[0].dWq,     [(i, j) for i in range(4) for j in range(4)]),
    ("block0.g1", model.blocks[0].gamma1, model.blocks[0].dgamma1, [(j,) for j in range(8)]),
    ("W_head",    model.W_head,           model.dW_head,           [(i, j) for i in range(4) for j in range(4)]),
]
worst, worst_name = 0.0, None
for name, P, dP, idxs in checks:
    analytic = np.array([dP[i] for i in idxs])
    numeric = numeric_grad_subset(model, xc, yc, P, idxs)
    e = rel_err(analytic, numeric)
    print(f"  {name:9s} max rel err {e:.3e}")
    if e > worst:
        worst, worst_name = e, name
print("WORST max rel err:", f"{worst:.3e}", "on", worst_name)

  E         max rel err 5.193e-09
  P         max rel err 3.992e-09
  block0.Wq max rel err 5.294e-05
  block0.g1 max rel err 4.847e-07
  W_head    max rel err 2.741e-08
WORST max rel err: 5.294e-05 on block0.Wq

Every sampled gradient matches finite differences, and the worst relative error across the entire model is 5.3e-05, comfortably under the 1e-4 bar. The pattern is instructive. The embeddings, the LM head, and the layer-norm scale check out at 10^{-7} to 10^{-9} because their gradients travel through relatively few operations. Wq is the noisiest at 5.3e-05 — still tiny — because at the small * 0.02 init the causal attention weights come out nearly uniform, so the query gradient is small and threads all the way back through the softmax in every head of both blocks, picking up the most finite-difference noise on the way. That every distinct part of the model gradient-checks is the numerical proof that backward implements the true derivative of forward for the whole GPT, not just for one block in isolation.

It Learns#

Matching finite differences proves the gradients are correct; the final check proves they are useful. We take one short fixed sequence — the first 17 characters of the corpus, "the lamp keeper w" — and ask the model to memorize it. Input is the first 16 characters, target is those 16 shifted by one. Then we run plain gradient descent: forward, backward, and subtract lr times every gradient, for a few hundred steps. If the gradients truly point downhill, a model with enough capacity to memorize one 16-character string should drive its loss from the log(24) baseline down toward zero.

model = MiniGPT(vocab_size, block_size, n_embd, n_head, n_layer, seed=42)
seq = data[:block_size + 1]
xo = seq[:block_size][None, :]                   # (1, 16) input
yo = seq[1:block_size + 1][None, :]              # (1, 16) next chars
print("sequence:", repr(decode(seq.tolist())))
lr = 0.3
for step in range(301):
    logits, loss = model.forward(xo, yo)
    model.backward()
    # SGD: nudge every parameter against its gradient
    model.E -= lr * model.dE
    model.P -= lr * model.dP
    model.gamma_f -= lr * model.dgamma_f
    model.beta_f -= lr * model.dbeta_f
    model.W_head -= lr * model.dW_head
    model.b_head -= lr * model.db_head
    for blk in model.blocks:
        for n in ["gamma1", "beta1", "Wq", "Wk", "Wv", "Wo",
                  "gamma2", "beta2", "W1", "b1", "W2", "b2"]:
            setattr(blk, n, getattr(blk, n) - lr * getattr(blk, "d" + n))
    if step % 50 == 0:
        print(f"  step {step:3d}  loss {loss:.4f}")

sequence: 'the lamp keeper w'
  step   0  loss 3.1788
  step  50  loss 0.0158
  step 100  loss 0.0064
  step 150  loss 0.0039
  step 200  loss 0.0028
  step 250  loss 0.0021
  step 300  loss 0.0017

The loss starts at 3.1788 — right at the log(24) baseline, because the model begins by guessing uniformly — and plunges to 0.0017 by step 300, more than three orders of magnitude below where it began. A loss that small means the model now assigns almost all of its probability to the correct next character at every one of the 16 positions: it has memorized the sequence. That is exactly what should happen when you point a model with enough parameters at a single short target and let correct gradients do their work. Overfitting one example is the canonical smoke test for a new architecture — if the loss had stalled or climbed, some gradient in the assembled model would be wrong. It falls, cleanly, which means every part connects and every gradient points the right way. Run either stage a second time and, because everything is seeded, you get byte-for-byte identical numbers; this course has no nondeterminism to hide behind.

Practice Exercises#

Try these before checking the hints. They reuse the MiniGPT class and the setup you built above.

Exercise 1: Gradient-Check b_head and a Second Block#

The Stage 4 check sampled E, P, block 0’s Wq and gamma1, and W_head. Extend it to two more places: the head bias b_head (all 24 entries) and block 1’s Wv (a 4x4 slice). Confirm both stay under 1e-4, so the check covers the head bias and the second block, not just the first.

# Your code here: add ("b_head", ...) and ("block1.Wv", ...) to the checks list

Exercise 2: Confirm Masked Positions Get No Embedding Gradient#

The causal mask means position 0 attends only to itself, so tokens that appear only at later positions still influence earlier ones through the loss — but a character id that never appears in the batch at all should get a zero row in dE. Verify this: after a forward and backward on xc, check that every row of dE for a character id not present in xc is exactly zero.

# Your code here: find ids missing from xc and assert their dE rows are all zero

Exercise 3: Overfit a Longer Sequence#

Rerun the Stage 4 overfit test on the full block_size window using a different slice of the corpus — say seq = data[500:500 + block_size + 1] — and confirm the loss again falls far below log(24). Does a different 16-character target reach a similarly tiny loss?

# Your code here: rebuild xo, yo from data[500:...] and rerun the SGD loop

Summary#

You assembled every component the course has built into one working GPT, and you proved it correct end to end. Let’s review what you built.

Key Concepts#

The Class Design

• MiniGPT(vocab_size, block_size, n_embd, n_head, n_layer) owns a token embedding E (24, 32), a position table P (16, 32), n_layer causal TransformerBlocks, a final LayerNorm (gamma_f, beta_f), and the LM head W_head (32, 24) with b_head
• The model fixes only widths, never B or T, and the head size falls out as n_embd // n_head = 8

forward(idx, targets)

• Adds token and position embeddings (E[idx] + P[:T]), runs the causal stack, applies the final norm, and projects to (B, T, 24) logits
• With targets, returns the next-token cross-entropy loss; a fresh model reports a loss near log(24) = 3.178 because it guesses uniformly over 24 characters

backward()

• Flows the loss back in reverse: dlogits = (probs - onehot) / N at the head, then final LayerNorm, then each block via its own backward (using dQuery, dKey, dValue inside attention), then the embeddings
• E’s gradient is a scatter-add (np.add.at) because many positions read the same row; P’s gradient sums over the batch

Proving Correctness

• A float64 gradient check across a slice of E, P, a block’s Wq and gamma1, and W_head gave a worst relative error of 5.3e-05, all under 1e-4
• Overfitting one fixed 16-character sequence drove the loss from 3.1788 to 0.0017 in 300 gradient-descent steps
• Every result is seeded and byte-for-byte reproducible

Why This Matters#

This is the model. Everything you spent six modules building — scaled dot-product attention, multiple heads, positional embeddings, residuals and layer norm and the feed-forward network, the causal mask, the language-modeling head and loss — now lives inside one MiniGPT object whose forward and backward you can trust because you gradient-checked the whole thing at once. Nothing about it is a black box: you know what every array is for and why every gradient has the shape it does.

That trust is what makes the next step possible. Training a neural network is nothing more than calling forward and backward in a loop and nudging the parameters — the exact two methods you just verified. Because the assembled model already overfits a single sequence, you know its gradients are correct and its capacity is real; scaling the same loop to the full corpus with a proper optimizer is a matter of more data and more steps, not new ideas. The hard part, building a provably correct GPT from nothing but NumPy, is behind you.

Continue Building Your Skills#

You now hold a complete, gradient-checked GPT in a single class — the same MiniGPT you will train for real in the next module. So far you have only proven that it can learn, by memorizing one short sequence; you have not yet taught it the language of Lantern Bay. In Module 7, Training the Tiny GPT, you will wrap this exact model in a real training loop: batching the full 10,020-character corpus into (B, T) windows, replacing plain gradient descent with an Adam optimizer that adapts each parameter’s step, and running for thousands of iterations while the loss falls from log(24) toward something genuinely low. You will watch the model move from uniform guessing to confident, context-aware character prediction — and because the forward and backward passes are already verified here, every bit of that progress will come from training, not from a bug you have to chase. Keep this MiniGPT class close; it is about to come alive.