https://datatweets.com/courses/transformers-from-scratch/positional-encoding-and-embeddings/Recent content in Positional Encoding & Embeddings on DATATWEETSHugoenCopyright (c) 2026 DatatweetsFri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/positional-encoding-and-embeddings/lesson-1-why-attention-needs-position/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/positional-encoding-and-embeddings/lesson-1-why-attention-needs-position/Self-attention treats a sequence as an unordered set of vectors: permute the input positions and the output rows simply permute the same way. This lesson proves that permutation equivariance in NumPy with seed 42 (max difference 4.4e-16, essentially zero), shows why identical tokens produce identical output rows when there is no position signal, and then adds a distinct vector to each position to break the symmetry (max difference 3.50). That is the motivation for the token embeddings and positional encodings you build across this module.https://datatweets.com/courses/transformers-from-scratch/positional-encoding-and-embeddings/lesson-2-token-embeddings/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/positional-encoding-and-embeddings/lesson-2-token-embeddings/A token embedding is a learned lookup: the id i maps to row E[i] of an embedding table, mathematically identical to one-hot(i) @ E but computed by indexing. This lesson builds E for the 24-character Lantern Bay vocabulary, embeds a short id sequence into a (1, T, C) tensor, then derives the backward pass — a scatter-add where each row’s gradient sums the upstream gradient at every position that token appeared. Everything is proven with a float64 finite-difference gradient check, with a real max relative error near 4e-10.https://datatweets.com/courses/transformers-from-scratch/positional-encoding-and-embeddings/lesson-3-sinusoidal-positional-encoding/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/positional-encoding-and-embeddings/lesson-3-sinusoidal-positional-encoding/Implement sinusoidal_encoding(T, C) in pure NumPy, where each dimension is a sinusoid of a different wavelength. You will verify every position gets a unique bounded encoding in [-1, 1], watch the dot product between encodings fall as positions move apart, understand why relative offsets are a fixed linear rotation, and add the encoding to token embeddings while preserving the (T, C) shape.https://datatweets.com/courses/transformers-from-scratch/positional-encoding-and-embeddings/lesson-4-learned-positional-embeddings/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/positional-encoding-and-embeddings/lesson-4-learned-positional-embeddings/Learn the positional approach GPT-2 uses instead of sinusoidal encoding: a trainable table P of shape (block_size, C) whose row t is the embedding for position t. You will add P[:T] to token embeddings in pure NumPy, derive the slice-add backward pass for dP, verify it against a float64 numerical gradient, and see why a learned table cannot extrapolate past block_size.https://datatweets.com/courses/transformers-from-scratch/positional-encoding-and-embeddings/lesson-5-guided-project-embedding-layer-with-positions/Fri, 10 Jul 2026 09:00:00 +0200https://datatweets.com/courses/transformers-from-scratch/positional-encoding-and-embeddings/lesson-5-guided-project-embedding-layer-with-positions/In this guided project you fold everything from Module 4 into a single reusable NumPy class. You build an EmbeddingLayer that holds a token table E and a learned position table P, add a forward(idx) that gathers E[idx] and adds P[:T] to produce (B, T, C), add a backward(dout) that scatter-adds dE with np.add.at and sums dP over the batch, then prove it: a float64 finite-difference gradient check gives max relative errors around 2.5e-10 for dE and 4.7e-11 for dP, and the (B, T, C) output feeds straight into the self-attention head from Modules 2-3. Everything is seeded and byte-for-byte reproducible.