https://datatweets.com/courses/transformers-from-scratch/from-sequences-to-attention/Recent content in From Sequences to Attention on DATATWEETSHugoenCopyright (c) 2026 DatatweetsFri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/from-sequences-to-attention/lesson-1-the-limits-of-recurrence/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/from-sequences-to-attention/lesson-1-the-limits-of-recurrence/Before you build a transformer, feel the problem it solves. You’ll implement a tiny RNN forward pass in pure NumPy, watch it squeeze any-length sequence into one fixed-size hidden vector, and measure how a change to the very first token fades from a norm of 2.74 down to about 1e-6 thirty steps later. Then you’ll count why recurrence can’t be parallelized — and preview how attention fixes all three limits at once.https://datatweets.com/courses/transformers-from-scratch/from-sequences-to-attention/lesson-2-the-attention-idea/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/from-sequences-to-attention/lesson-2-the-attention-idea/Attention is the single most important idea in this course. This lesson introduces it as a soft dictionary lookup: instead of retrieving one value for an exact key, you compare a query against every key, softmax the similarities into weights, and return a weighted average of the values. You build a real NumPy demo (seed 42) with a tiny toy memory and watch a sharp query act like a near-hard lookup while a query between two keys blends their values.https://datatweets.com/courses/transformers-from-scratch/from-sequences-to-attention/lesson-3-similarity-scores-and-softmax/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/from-sequences-to-attention/lesson-3-similarity-scores-and-softmax/Build the machinery that converts vectors into attention weights. You will use dot-product similarity to score a query against several keys, turn those raw scores into a probability distribution with a numerically stable softmax, watch the naive version overflow to NaN while the stable one holds, and use temperature scaling to make the same weights sharper or softer — every step run for real in NumPy with seed 42.https://datatweets.com/courses/transformers-from-scratch/from-sequences-to-attention/lesson-4-from-lookup-to-learned-attention/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/from-sequences-to-attention/lesson-4-from-lookup-to-learned-attention/In Lessons 2 and 3 you built a content-based lookup using the input vectors directly as query, key, and value. That fixes what similarity means. This lesson adds the learned projections Wq, Wk, Wv that turn the input X into Q, K, and V in three separate learned spaces, so the model can learn from data which relationships matter. You implement the parameterized self-attention forward in NumPy, print the shape at every stage, confirm attention rows sum to 1, and watch a change to Wq alone rewire which positions attend to which.https://datatweets.com/courses/transformers-from-scratch/from-sequences-to-attention/lesson-5-guided-project-retrieval-attention-layer/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/from-sequences-to-attention/lesson-5-guided-project-retrieval-attention-layer/In this guided project you build the first working piece of the course’s tiny GPT. You tokenize the Lantern Bay corpus at the character level, embed a 12-character slice into an (1, 12, 32) tensor, run one learned query/key/value attention layer to get an (1, 12, 12) weight matrix, and inspect it — discovering that repeated characters produce identical attention rows because there is no positional signal yet. Everything runs in pure NumPy with seed 42.