Generation & Sampling

Welcome to Generation & Sampling, the eighth module. A trained language model doesn’t produce text — it produces, at every step, a probability distribution over the next token. Turning that distribution into an actual sequence of characters is a separate decision, and the strategy you choose changes everything about how the output reads: repetitive or varied, coherent or unhinged. This module is about making that choice well.

You’ll start with greedy decoding — always taking the single most likely token — and see exactly why it produces flat, looping text. Then you’ll add controlled randomness: temperature to sharpen or soften the distribution, top-k sampling to draw only from the k most likely tokens and cut off the unlikely tail, and top-p (nucleus) sampling to keep just enough of the distribution’s mass to stay coherent while adapting to how confident the model is. Each is a few lines of NumPy on top of the logits your trained mini-GPT already produces. The guided project pulls them together into a single, configurable text generator, and you’ll compare the same model’s output across settings to feel what each knob does.

Start with Lesson 1, where the safest possible decoding strategy turns out to be a trap.