Self-Attention from Scratch

Welcome to Self-Attention from Scratch, the second module — where the intuition from Module 1 becomes a real, trainable layer. By the end you’ll have implemented the exact operation at the core of every transformer, both its forward pass and its backward pass, and proven your gradients are correct.

You’ll start by formalizing the three learned projections — query, key, and value — that let a sequence attend to itself. Then you’ll build scaled dot-product attention in full and run a real experiment showing why the $1/\sqrt{d}$ scaling isn’t optional: without it, dot products grow with dimension and softmax saturates into a near one-hot spike that kills gradients. You’ll learn to read the attention matrix as a set of per-position distributions, and then take on the hardest and most rewarding part of the course so far: deriving the backward pass through attention by hand — through the value multiply, the softmax, and the score matrix — and checking every gradient against a finite-difference estimate. The guided project assembles a reusable, gradient-checked self-attention layer you’ll stack in later modules.

Start with Lesson 1, where the query, key, and value projections get their precise definitions.