Title here
Summary here
Lesson 5 - Guided Project: A Text Generator
Welcome to the Guided Project
Over the last four lessons you built up a toolbox of decoding strategies one at a time: greedy argmax, temperature scaling, top-k truncation, and top-p (nucleus) filtering. Each lived in its own small function so you could study it in isolation. That is great for learning, but nobody ships four near-duplicate generators. In this guided project you consolidate everything into one configurable generate function on your trained MiniGPT, the kind of function a real sampling API exposes: pass a prompt and a handful of knobs, and the same code path handles deterministic greedy decoding, plain temperature sampling, top-k, and nucleus sampling.
This is the capstone of the module. You will train the char-level model on the Lantern Bay corpus, wire up the unified generator, and then do the thing that actually matters: run the same prompt through several settings and read the real output honestly. On this tiny low-entropy corpus the differences are subtle but genuine, and seeing them on text your own model produced is worth more than any table of formulas.
By the end of this project, you will be able to:
- Train (or reload) the from-scratch
MiniGPTand confirm its loss has settled near 0.13 - Implement one
generate(model, prompt, n_new, temperature, top_k, top_p, greedy, rng)that covers greedy, temperature, top-k, and top-p in a single function - Apply temperature, then an optional top-k filter, then an optional top-p filter, in the correct order before renormalizing and sampling
- Compare decoding strategies on identical prompts and describe how each reads
- Reason about what is exactly reproducible (greedy, and any seeded sample) versus what varies with the random seed
You should have the module’s earlier lessons behind you: the greedy loop from Lesson 1, temperature from Lesson 2, top-k from Lesson 3, and top-p from Lesson 4. Everything here is pure NumPy, no framework and no API key. Let’s build it.
Stage 1: Train the model
Everything in this project runs on a real trained model, so we start by training one. This is the same char-level MiniGPT you built through Modules 1 to 7: token and position embeddings, two transformer blocks with causal multi-head attention and a feed-forward network, a final layer norm, and a linear head to vocabulary logits. We reuse the canonical Lantern Bay corpus and the Adam optimizer, and train for about 1200 steps at a learning rate of 3e-3. On CPU this finishes in well under half a minute.
We include the full model here so the project is self-contained and runnable end to end. If you saved weights from Module 7, you could load them instead with np.load; training inline just keeps everything in one place.
import numpy as np, time, warnings
warnings.filterwarnings("ignore")
# ---------- corpus (canonical Lantern Bay) ----------
CORPUS = (
"the lamp keeper walks the shore at dusk. "
"the lamp keeper lights the lantern when the fog rolls in. "
"the boats come home when the lantern glows. "
"the boats wait outside the bay when the fog is thick. "
"a small boat drifts near the rocks at dawn. "
"the keeper rings the bell when a boat drifts near the rocks. "
"the tide rises at dusk and falls at dawn. "
"the gulls call over the bay when the tide is low. "
"the keeper counts the boats and writes the count in a book. "
"the lantern burns all night and rests at dawn. "
) * 20
chars = sorted(set(CORPUS)); vocab_size = len(chars)
stoi = {c: i for i, c in enumerate(chars)}
itos = {i: c for i, c in enumerate(chars)}
encode = lambda s: [stoi[c] for c in s]
decode = lambda ids: "".join(itos[i] for i in ids)
data = np.array(encode(CORPUS), dtype=np.int64)
def softmax(x, axis=-1):
x = x - x.max(axis=axis, keepdims=True)
e = np.exp(x)
return e / e.sum(axis=axis, keepdims=True)
def get_batch(dat, block_size, batch_size, rng):
ix = rng.integers(0, len(dat) - block_size - 1, size=batch_size)
x = np.stack([dat[i:i+block_size] for i in ix])
y = np.stack([dat[i+1:i+block_size+1] for i in ix])
return x, y
class Adam:
def __init__(self, params, lr=3e-3, b1=0.9, b2=0.999, eps=1e-8):
self.lr, self.b1, self.b2, self.eps = lr, b1, b2, eps; self.t = 0
self.m = {k: np.zeros_like(v) for k, v in params.items()}
self.v = {k: np.zeros_like(v) for k, v in params.items()}
def step(self, params, grads):
self.t += 1
for k in params:
g = grads[k]
self.m[k] = self.b1*self.m[k] + (1-self.b1)*g
self.v[k] = self.b2*self.v[k] + (1-self.b2)*(g*g)
mh = self.m[k]/(1-self.b1**self.t); vh = self.v[k]/(1-self.b2**self.t)
params[k] -= self.lr*mh/(np.sqrt(vh)+self.eps)
class MiniGPT:
def __init__(self, vocab_size, block_size, n_embd, n_head, n_layer, seed=42):
rng = np.random.default_rng(seed)
self.cfg = (vocab_size, block_size, n_embd, n_head, n_layer)
self.n_head = n_head; self.n_layer = n_layer; self.hs = n_embd // n_head
P = {}; s = 0.02
P['E'] = rng.normal(0, s, (vocab_size, n_embd))
P['P'] = rng.normal(0, s, (block_size, n_embd))
for l in range(n_layer):
P[f'ln1g{l}'] = np.ones(n_embd); P[f'ln1b{l}'] = np.zeros(n_embd)
P[f'Wq{l}'] = rng.normal(0, s, (n_embd, n_embd))
P[f'Wk{l}'] = rng.normal(0, s, (n_embd, n_embd))
P[f'Wv{l}'] = rng.normal(0, s, (n_embd, n_embd))
P[f'Wo{l}'] = rng.normal(0, s, (n_embd, n_embd))
P[f'ln2g{l}'] = np.ones(n_embd); P[f'ln2b{l}'] = np.zeros(n_embd)
P[f'W1{l}'] = rng.normal(0, s, (n_embd, 4*n_embd)); P[f'b1{l}'] = np.zeros(4*n_embd)
P[f'W2{l}'] = rng.normal(0, s, (4*n_embd, n_embd)); P[f'b2{l}'] = np.zeros(n_embd)
P['lnfg'] = np.ones(n_embd); P['lnfb'] = np.zeros(n_embd)
P['Wh'] = rng.normal(0, s, (n_embd, vocab_size)); P['bh'] = np.zeros(vocab_size)
self.P = P
def n_params(self): return sum(v.size for v in self.P.values())
def _ln_f(self, x, g, b):
mu = x.mean(-1, keepdims=True); xc = x - mu
var = (xc**2).mean(-1, keepdims=True); std = np.sqrt(var + 1e-5)
xh = xc/std; return g*xh + b, (xh, std, g)
def _ln_b(self, dout, cache):
xh, std, g = cache; C = xh.shape[-1]; dxh = dout*g
dx = (1/(C*std))*(C*dxh - dxh.sum(-1, keepdims=True) - xh*(dxh*xh).sum(-1, keepdims=True))
return dx, (dout*xh).reshape(-1, C).sum(0), dout.reshape(-1, C).sum(0)
def forward(self, idx, targets=None):
V, BS, C, H, L = self.cfg; B, T = idx.shape; P = self.P; hs = self.hs; ca = {}
x = P['E'][idx] + P['P'][:T]; ca['idx'] = idx; ca['T'] = T
mask = np.triu(np.ones((T, T), dtype=bool), k=1)
for l in range(L):
ln1, c1 = self._ln_f(x, P[f'ln1g{l}'], P[f'ln1b{l}'])
Q = (ln1 @ P[f'Wq{l}']).reshape(B, T, H, hs).transpose(0, 2, 1, 3)
K = (ln1 @ P[f'Wk{l}']).reshape(B, T, H, hs).transpose(0, 2, 1, 3)
Vv = (ln1 @ P[f'Wv{l}']).reshape(B, T, H, hs).transpose(0, 2, 1, 3)
sc = Q @ K.transpose(0, 1, 3, 2) / np.sqrt(hs)
sc = np.where(mask, -1e9, sc); A = softmax(sc, -1)
ho = (A @ Vv).transpose(0, 2, 1, 3).reshape(B, T, C); att = ho @ P[f'Wo{l}']
x2 = x + att
ln2, c2 = self._ln_f(x2, P[f'ln2g{l}'], P[f'ln2b{l}'])
h = ln2 @ P[f'W1{l}'] + P[f'b1{l}']; hr = np.maximum(h, 0)
ff = hr @ P[f'W2{l}'] + P[f'b2{l}']
x = x2 + ff
ca[l] = (ln1, c1, Q, K, Vv, A, ho, x2, ln2, c2, h, hr)
xf, cf = self._ln_f(x, P['lnfg'], P['lnfb']); logits = xf @ P['Wh'] + P['bh']
ca['xf'] = xf; ca['cf'] = cf; self.ca = ca
if targets is None: return logits, None
B, T, Vt = logits.shape; flat = logits.reshape(-1, Vt); tgt = targets.reshape(-1)
p = softmax(flat, -1); loss = -np.log(p[np.arange(len(tgt)), tgt] + 1e-12).mean()
self.ca['p'] = p; self.ca['tgt'] = tgt; self.ca['shape'] = (B, T, Vt)
return logits, loss
def backward(self):
V, BS, C, H, L = self.cfg; P = self.P; hs = self.hs; ca = self.ca
g = {k: np.zeros_like(v) for k, v in P.items()}
p = ca['p'].copy(); tgt = ca['tgt']; B, T, Vt = ca['shape']; N = len(tgt)
p[np.arange(N), tgt] -= 1; dlogits = (p/N).reshape(B, T, Vt)
xf = ca['xf']
g['Wh'] = xf.reshape(-1, C).T @ dlogits.reshape(-1, Vt)
g['bh'] = dlogits.reshape(-1, Vt).sum(0)
dxf = dlogits @ P['Wh'].T
dx, g['lnfg'], g['lnfb'] = self._ln_b(dxf, ca['cf'])
mask = np.triu(np.ones((T, T), dtype=bool), k=1)
for l in reversed(range(L)):
ln1, c1, Q, K, Vv, A, ho, x2, ln2, c2, h, hr = ca[l]
dx2 = dx.copy(); dff = dx
g[f'W2{l}'] = hr.reshape(-1, 4*C).T @ dff.reshape(-1, C)
g[f'b2{l}'] = dff.reshape(-1, C).sum(0)
dhr = dff @ P[f'W2{l}'].T; dh = dhr*(h > 0)
g[f'W1{l}'] = ln2.reshape(-1, C).T @ dh.reshape(-1, 4*C)
g[f'b1{l}'] = dh.reshape(-1, 4*C).sum(0)
dln2 = dh @ P[f'W1{l}'].T
dx2b, g[f'ln2g{l}'], g[f'ln2b{l}'] = self._ln_b(dln2, c2); dx2 = dx2 + dx2b
dx1 = dx2.copy(); datt = dx2
g[f'Wo{l}'] = ho.reshape(-1, C).T @ datt.reshape(-1, C)
dho = datt @ P[f'Wo{l}'].T
dHe = dho.reshape(B, T, H, hs).transpose(0, 2, 1, 3)
dA = dHe @ Vv.transpose(0, 1, 3, 2); dValue = A.transpose(0, 1, 3, 2) @ dHe
dsc = A*(dA - (dA*A).sum(-1, keepdims=True)); dsc = np.where(mask, 0, dsc)/np.sqrt(hs)
dQuery = dsc @ K; dKey = dsc.transpose(0, 1, 3, 2) @ Q
def merge(t): return t.transpose(0, 2, 1, 3).reshape(B, T, C)
dQm, dKm, dVm = merge(dQuery), merge(dKey), merge(dValue)
g[f'Wq{l}'] = ln1.reshape(-1, C).T @ dQm.reshape(-1, C)
g[f'Wk{l}'] = ln1.reshape(-1, C).T @ dKm.reshape(-1, C)
g[f'Wv{l}'] = ln1.reshape(-1, C).T @ dVm.reshape(-1, C)
dln1 = dQm @ P[f'Wq{l}'].T + dKm @ P[f'Wk{l}'].T + dVm @ P[f'Wv{l}'].T
dx1b, g[f'ln1g{l}'], g[f'ln1b{l}'] = self._ln_b(dln1, c1); dx1 = dx1 + dx1b
dx = dx1
np.add.at(g['E'], ca['idx'], dx); g['P'][:T] = dx.sum(axis=0)
return g
def train_model(steps=1201, lr=3e-3, seed=42):
rng = np.random.default_rng(seed)
model = MiniGPT(vocab_size, 32, 64, 4, 2, seed=42)
opt = Adam(model.P, lr=lr)
n = int(len(data)*0.9); tr = data[:n]
t0 = time.time()
for step in range(steps):
xb, yb = get_batch(tr, 32, 32, rng)
_, loss = model.forward(xb, yb); g = model.backward(); opt.step(model.P, g)
if step % 200 == 0 or step == steps - 1:
print(f" step {step:4d} train loss {loss:.4f}")
print(f" trained in {time.time()-t0:.1f}s final loss {loss:.4f}")
return model
print("vocab_size", vocab_size, "corpus len", len(data))
model = train_model()Running this prints the training curve and the final loss:
vocab_size 24 corpus len 10020
step 0 train loss 3.1948
step 200 train loss 0.1924
step 400 train loss 0.1583
step 600 train loss 0.1344
step 800 train loss 0.1291
step 1000 train loss 0.1621
step 1200 train loss 0.1326
trained in 18.1s final loss 0.1326The loss falls from about 3.19 (close to log(24) = 3.18, the entropy of a uniform guess over 24 characters) to about 0.13 in under 20 seconds. That is a model that has essentially memorized the structure of this tiny corpus, which is exactly what we want for studying decoding: the probability distributions are sharp and confident, so the effect of each sampling knob is easy to see.
The seed makes training reproducible
Both the weight initialization (MiniGPT(..., seed=42)) and the batch sampler (np.random.default_rng(seed)) are seeded, so this training run is fully deterministic. Re-run the cell and you get the identical loss curve and the identical trained weights, character for character. Every number in this lesson was produced by running the code exactly as printed.
Stage 2: The unified generator
Now the heart of the project. We want a single function that subsumes all four strategies from the module. The design principle is that these strategies are not alternatives to each other, they are filters applied in sequence to the final-position logits. Temperature reshapes the whole distribution; top-k keeps only the k highest-scoring tokens; top-p keeps the smallest set of tokens whose probabilities sum to at least p. You can stack them, and greedy decoding is simply the special case where you skip all of that and take the argmax.
Here is the order of operations for one step, applied to logits[0, -1], the logits for the token that comes after the current context:
- If
greedy, takeargmaxand stop, no randomness, no filters. - Otherwise divide the logits by
temperature. - If
top_kis set, keep only theklargest logits, set the rest to . - If
top_pis set, sort by probability, keep the smallest prefix whose cumulative mass reachestop_p, set the rest to . - Softmax the survivors so they renormalize to a proper distribution, then sample one index with the provided random generator.
The function stays close to the individual versions you already wrote; it just guards each filter behind an if and threads a single rng through for reproducible sampling.
def generate(model, prompt, n_new, temperature=1.0, top_k=None, top_p=None,
greedy=False, rng=None):
"""Autoregressively extend `prompt` (a string) by `n_new` characters.
One function covering greedy / temperature / top-k / top-p decoding."""
if rng is None:
rng = np.random.default_rng(0)
block_size = model.cfg[1]
ids = list(encode(prompt))
for _ in range(n_new):
ctx = np.array([ids[-block_size:]]) # (1, T<=block_size)
logits, _ = model.forward(ctx) # (1, T, vocab)
row = logits[0, -1].astype(np.float64) # (vocab,) last position
if greedy: # deterministic argmax
ids.append(int(row.argmax()))
continue
row = row / temperature # temperature scaling
if top_k is not None: # top-k filter (Lesson 3)
k = min(top_k, row.shape[0])
keep = np.argpartition(row, -k)[-k:]
filt = np.full_like(row, -np.inf)
filt[keep] = row[keep]
row = filt
if top_p is not None: # top-p / nucleus (Lesson 4)
order = np.argsort(row)[::-1] # high -> low logit
probs = softmax(row)[order]
csum = np.cumsum(probs)
cutoff = np.searchsorted(csum, top_p) + 1 # smallest set with mass >= top_p
drop = order[cutoff:]
row[drop] = -np.inf
probs = softmax(row) # renormalize survivors
ids.append(int(rng.choice(len(probs), p=probs)))
return decode(ids)A few details worth pausing on. We cast the logit row to float64 before filtering so that setting entries to -np.inf and running them through exp is clean (those become exactly zero probability). We apply the filters to logits, not probabilities, so a single final softmax does the renormalization for us, whether we filtered or not. And greedy=True short-circuits before temperature even runs, because dividing by temperature never changes which logit is largest, so greedy decoding ignores every other knob. That is why the diagram draws greedy as a dashed bypass around the whole pipeline.
One rng in, reproducible samples out
The generator never calls the global NumPy random state; it only draws from the rng you pass in. That means a sample is reproducible exactly when you hand it the same seeded generator, for example rng=np.random.default_rng(42). Greedy decoding takes no draws at all, so it is reproducible unconditionally. This separation is what lets the comparison below be honest: the greedy line is fixed forever, and each sampled line is fixed to its seed.
Stage 3: Compare strategies
Now we run the same prompt, "the ", through several settings and read the results. Each sampled call gets a freshly seeded generator so the output is reproducible.
prompt = "the "
print("prompt:", repr(prompt))
print("greedy :", repr(generate(model, prompt, 80, greedy=True)))
print("temperature=0.8:", repr(generate(model, prompt, 80, temperature=0.8,
rng=np.random.default_rng(42))))
print("top_k=5 :", repr(generate(model, prompt, 80, top_k=5,
rng=np.random.default_rng(42))))
print("top_p=0.9 :", repr(generate(model, prompt, 80, top_p=0.9,
rng=np.random.default_rng(42))))
print("temperature=1.5:", repr(generate(model, prompt, 80, temperature=1.5,
rng=np.random.default_rng(42))))The real output:
prompt: 'the '
greedy : 'the boats and writes the count in a book. the lantern burns all night and rests at d'
temperature=0.8: 'the lantern burns all night and rests at dawn. the lamp keeper walks the shore at du'
top_k=5 : 'the lantern burns all night and rests at dawn. the lamp keeper walks the shore at du'
top_p=0.9 : 'the lantern burns all night and rests at dawn. the lamp keeper walks the shore at du'
temperature=1.5: 'the rockse at dawn. the keeper rings the bell when a boat drifts near the rocks at d'Read these honestly, because the result is more interesting than a tidy “each setting looks different” claim would be. Greedy reproduces a stretch of the corpus verbatim: it deterministically walks the single most likely path, which on a memorized corpus is a real sentence stitched together, well-formed but rigid. It is the same every time you run it.
The three moderate sampling settings, temperature 0.8, top_k=5, and top_p=0.9, all produce the same coherent continuation here, and different from greedy. That coincidence is not a bug, it is a lesson about this corpus. The trained model is extremely confident: at most positions one character carries almost all the probability mass, so scaling by 0.8, keeping the top 5, or keeping 90% of the mass all leave the same handful of near-certain characters in play, and the seeded sampler then walks the same path. On a real, high-entropy corpus these three would diverge, because the tail the model is uncertain about would be much fatter.
Temperature 1.5 is where the difference finally shows. Flattening the distribution gives low-probability characters a real chance, and the model slips: "the rockse at dawn" is not anything in the corpus, "rockse" is a small spelling error the model would never commit at low temperature. This is the honest tradeoff of the whole module in one line: higher temperature buys variety at the cost of mistakes, lower temperature buys coherence at the cost of variety.
We can also verify the reproducibility claims directly:
a = generate(model, prompt, 80, greedy=True)
b = generate(model, prompt, 80, greedy=True)
print("greedy identical twice:", a == b)
s1 = generate(model, prompt, 40, temperature=0.8, top_k=10, rng=np.random.default_rng(7))
s2 = generate(model, prompt, 40, temperature=0.8, top_k=10, rng=np.random.default_rng(7))
print("sampling identical with same seed:", s1 == s2)greedy identical twice: True
sampling identical with same seed: TrueAnd that sampling genuinely varies when you change the seed, which is the flip side of the same coin:
for seed in (1, 7, 99):
print("seed", seed, ":",
repr(generate(model, "the ", 60, temperature=1.0, top_k=8,
rng=np.random.default_rng(seed))))seed 1 : 'the keeper rings the bell when a boat drifts near the rocks at d'
seed 7 : 'the lantern when the fog rolls in. the boats come home when the '
seed 99 : 'the keeper rings the bell when a boat drifts near the rocks. the'Three seeds, three different openings from the same prompt and the same settings. Sampling is not deterministic; it is deterministic given a seed. That is the property you rely on when you want a demo to be reproducible without pinning it to greedy’s single rigid output.
Stage 4: A short generated passage
To close the module, let’s generate one longer passage from the finished model with sensible everyday settings: temperature 0.8 to keep it coherent, and top_k=10 to trim the tail so a stray low-probability character cannot derail a whole sentence. We seed the generator so the passage is reproducible, and prompt with "the keeper ".
passage = generate(model, "the keeper ", 200, temperature=0.8, top_k=10,
rng=np.random.default_rng(42))
print(repr(passage))
print("length:", len(passage))'the keeper rings the bell when a boat drifts near the rocks. the tide rises at dusk and falls at dawn. the gulls call over the bay when the tide is low. the keeper counts the boats and writes the count in a book'
length: 211That is a genuine sample from a transformer you built from nothing but NumPy: embeddings, attention, the causal mask, layer norm, a feed-forward network, a hand-derived backward pass, an Adam training loop, and now a full decoding stack. It reads as clean, grammatical Lantern Bay prose. Be honest about what it is, though: this is a char-level model that has largely memorized a tiny, repetitive corpus, so it recombines familiar sentences rather than inventing new ideas. Change the seed and you get a different but equally in-distribution passage. Scale the same architecture up, feed it a large diverse corpus, and this exact decoding function is what turns the learned distribution into fluent text.
The generator is architecture-agnostic
Nothing in generate knows how big the model is. It only needs model.forward to return next-token logits and model.cfg[1] for the context length. The same function that samples from this 154k-parameter char model would sample from a model a thousand times larger. Decoding and the model are cleanly separated, which is exactly why production systems ship one configurable generator rather than four.
Practice Exercises
Exercise 1: Add a repetition_penalty. Extend generate with a repetition_penalty=1.0 argument. Before sampling, divide the logit of every character already present in ids by the penalty when its logit is positive, and multiply when negative (the standard scheme), so repeated characters become less likely. Generate 120 characters with penalty 1.0 versus 1.3 at temperature=1.0 and describe whether the looping you saw with greedy decoding eases.
Hint
Work on the row of logits after temperature scaling but before the top-k and top-p filters. Get the set of used ids with set(ids), and for each t in it apply row[t] = row[t] / p if row[t] > 0 else row[t] * p. Because a positive logit shrinks toward zero and a negative one grows more negative, both directions reduce that character’s probability.
Exercise 2: Combine top-k and top-p. Call generate with both top_k=8 and top_p=0.9 set at once, and confirm from the code that top-k runs first (capping the candidate set to 8) and top-p then trims within that capped set. Print a 60-character sample and compare it to using either filter alone with the same seed. Explain why applying top-k first can only ever remove candidates that top-p might otherwise have kept.
Hint
Trace the order in the function: top_k sets everything outside the k best to first, so by the time the top_p block sorts and takes a cumulative-mass prefix, at most k finite logits remain. The nucleus is therefore chosen from the survivors of top-k, never from the full vocabulary. Use np.random.default_rng(0) for all three calls so the only variable is the filter.
Exercise 3: Measure how often greedy loops. Greedy decoding is deterministic, so once it enters a repeated context it repeats forever. Generate 400 greedy characters from "the ", then scan the output for the first index i where a 20-character window text[i:i+20] reappears later in the string. Report that index as a rough “loop onset” and explain why sampling with top_k=10 does not have a fixed loop point.
Hint
A simple detector: for each start i, check text[i:i+20] in text[i+20:]. The first i that returns True marks where greedy’s output starts recycling earlier text. Sampling breaks the strict determinism, so even after revisiting a context the next character can differ, which is exactly the repetition problem greedy has and sampling avoids.
Summary
You consolidated the entire module into one generate function and put it to work on a real trained model. You trained the char-level MiniGPT on the Lantern Bay corpus to a loss near 0.13, then built a single generator that forwards the last block_size tokens, applies temperature, an optional top-k filter, and an optional top-p filter in that order, renormalizes with a softmax, and samples, or shortcuts to argmax when greedy=True. You compared greedy, temperature, top-k, and nucleus settings on the same prompt, saw that a confident low-entropy model makes moderate filters read alike while high temperature introduces real errors, and finished by generating a coherent 200-character passage from your own from-scratch transformer.
Key Concepts
- Filters compose in sequence. Temperature, top-k, and top-p are not competing strategies but stackable transforms on the final-position logits, applied before a single renormalizing softmax.
- Greedy is the no-randomness special case. It bypasses temperature and every filter and takes the argmax, so it is fully deterministic and reproducible without a seed.
- Filter on logits, sample once. Setting excluded logits to and applying one softmax at the end handles renormalization uniformly, whether or not any filter ran.
- Reproducibility is seed-scoped. A sample is exactly reproducible given the same
rng, and varies across seeds; greedy needs no seed to be reproducible. - Decoding is separate from the model. The generator only needs next-token logits and the context length, so the same function scales from this tiny model to any larger one.
Why This Matters
Every text-generating system you will ever use, from an autocomplete box to a chat assistant, sits behind a function shaped almost exactly like this one: a trained model that emits next-token logits, and a decoding layer with temperature, top-k, and top-p knobs. Having written that layer yourself in a few dozen lines of NumPy, you understand precisely what those knobs do, why a “temperature” slider makes output wilder, and why two runs at the same settings can still differ unless the seed is pinned. That is not a metaphor for how sampling works; it is the mechanism, and you now own it end to end.
Continue Building Your Skills
You have reached the finish line of the generation module with a complete, configurable text generator running on a transformer you built from scratch. The next module is the course capstone, Module 9, where you bring the whole journey together: assembling the full model, training and decoding pipeline into one coherent project, reflecting on how the pieces you built module by module, embeddings, attention, the causal mask, layer norm, backpropagation, training, and this decoding stack, form a working miniature GPT, and considering how each design choice would change as you scale the architecture up. Take a moment to re-run this generator with your own prompts and settings before you move on; the intuition you build by playing with temperature and the filters is exactly what the capstone asks you to draw on.
Previous lesson
Lesson 4 - Top-p (Nucleus) SamplingModule overview
Back to Module Overview