Volume V · Respect the Silicon

Never Build the Grid

A cook with a small cutting board and a large pantry does not carry the pantry to the board. Every previous volume changed what attention computes. This one changes only where the numbers sit while it computes them — and that turned out to matter more.

The bill nobody was reading

A GPU datasheet lists two headline numbers, and for a decade the field read only one of them. The first is arithmetic throughput: how many multiply-adds per second. The second is memory bandwidth: how fast numbers can be moved between the large off-chip memory and the small fast memory beside the arithmetic units. The two have not improved at the same rate. Arithmetic ran ahead, bandwidth lagged, and the gap widened every generation until a great many kernels stopped being limited by how fast they could compute and started being limited by how fast they could fetch.

Attention is one of them. This is the reframing that opens the volume, and it is worth stating flatly because it contradicts how every earlier lecture has counted: attention is memory-bandwidth-bound, not FLOP-bound. Its arithmetic is a pair of matrix multiplications, which modern silicon eats. Its sin is elsewhere. The textbook implementation computes the n × n score matrix, writes it out to high-bandwidth memory, reads it back to apply the softmax, writes the result out again, and reads it back once more to multiply by V. Four journeys across the slowest link in the machine, carrying the largest object in the calculation.

One hundred and thirty-four megabytes

Put a number on it, because the number is what lands the point. Take a sequence of 8192 tokens and one attention head, with scores held in 16-bit precision. The score matrix is 8192 × 8192 entries at two bytes each: 8192² × 2 ≈ 134 megabytes. That object is written to memory and read back — per head, per layer, on every forward pass. Multiply by the heads, multiply by the layers, and the traffic is not a detail of the implementation. It is the implementation.

Tiling, and the trick that permits it

FlashAttention, published by Dao and colleagues in 2022, rests on two ideas. The first is tiling: split Q, K and V into blocks small enough to fit in on-chip SRAM, and process one block-pair at a time — load the tile, compute its scores, use them immediately, discard them. The score matrix is never assembled anywhere. It exists only as a succession of small tiles that live and die on the chip.

The obvious objection is softmax, and it is a good one. Softmax normalises across a whole row: to know the denominator you must have seen every score in that row. Tiling shows you a few at a time. On the face of it the two are incompatible, and this incompatibility is why the naive implementation materialises the grid in the first place — it is not laziness, it is the algorithm asking for the whole row.

The second idea removes the requirement. Online softmax maintains two running quantities as tiles arrive: the maximum score seen so far, and the sum of exponentials so far. When a new tile arrives with a larger maximum, the accumulated sum and the accumulated output are rescaled by the appropriate factor and the new tile is folded in. At the end, the accumulated output divided by the accumulated sum is exactly what a full-row softmax would have produced. Not an approximation of it — the same number, to the same precision the arithmetic allows.

Add the two together and the accounting changes class. The arithmetic is unchanged: still O(n²) multiply-adds, not one of them skipped. The memory is now O(n) rather than O(n²), because nothing of quadratic size is ever stored. And the traffic across the slow link falls by a large constant factor, because Q, K and V are streamed through once instead of a quadratic object being written and read repeatedly.

The pantry and the cutting board

The kitchen analogy is worth keeping, because it makes the cost model intuitive. High-bandwidth memory is the pantry: large, and a walk away. SRAM is the cutting board: tiny, and directly under your hands. The naive method hauls the entire pantry out onto the counter for every dish, uses one ingredient, and carries it all back. FlashAttention brings out one tray at a time, works it completely, and fetches the next.

Note what the analogy does not claim. The cook does not chop fewer vegetables. The recipe is unchanged and the meal is identical. The only thing that changed is the walking, and the walking was most of the evening.

Two more versions

FlashAttention-2 (2023) is not a new algorithm but a better one on the same idea: improved partitioning of work across thread blocks and warps, reducing non-matmul operations and idle units. The reported result is roughly a 2× speedup over the first version, reaching 50–73% of theoretical peak throughput on A100.

FlashAttention-3 (Shah et al., 2024) targets the Hopper generation specifically, and its gains come from asynchrony — overlapping data movement with computation so the memory pipeline and the arithmetic units are busy at once — together with support for FP8. The reported result is roughly 1.5–2× over FlashAttention-2, at around 75% utilisation on H100.

The pattern across the three is instructive on its own. Version one found the idea. Versions two and three are almost entirely about fitting that idea to the particular shape of a particular chip. That is what "respect the silicon" means as a discipline: the algorithm was settled in 2022 and the performance work has continued ever since, generation by generation.

What it did to the argument

Now connect this back to Volume III, because FlashAttention quietly changed the terms of that debate. The sparse and linear families were motivated by two costs at once: quadratic arithmetic and quadratic memory. FlashAttention removed the second one entirely, for exact attention, without discarding a single multiplication. Full attention's memory footprint stopped being quadratic in 2022.

That is why exact attention stayed competitive for so much longer than the 2020 papers expected. The efficient variants suddenly had to justify themselves on arithmetic alone — against a rival that was now well engineered, numerically exact, and running near the hardware's limit. Every efficiency claim in the preceding volumes is measured against that baseline, and it is a far higher bar than the naive implementation those papers were originally arguing with.

The next lecture keeps the same posture and moves one level out. FlashAttention fixed how attention uses memory during a single forward pass. It said nothing about how a serving engine manages the memory of thousands of concurrent requests — which turns out to be a problem operating systems solved in the 1960s.