A lecture series · Revised July 2026
A 2026 technical report reads like a foreign language until one part of it becomes fluent. This series starts with that one part — attention, a single machine with a single argument — and only once it is second nature does it widen: how the experts are chosen, what precision the weights are kept in, how training survives trillion-token runs without exploding, and what happens to a model after pretraining ends.
Scroll slowly ↓
Start with attention, because it is one idea told four ways. Everything after it is the same kind of story — a cost, and the mental models invented to pay it down.Part One (Volumes I–VI) is the argument over how a model remembers. By 2026 every lab agreed on the mixture-of-experts skeleton and then disagreed, in public, about attention — DeepSeek and GLM bet on trainable sparsity, Kimi and Qwen on linear hybrids, MiniMax tried linear, retreated to full, then shipped sparse. Parts Two through Four take the same instinct to how the experts are picked, what precision the weights are stored at, and how a model is trained and then re-trained after the fact. Fourteen lectures on attention; ten on everything the report says next.
Comparing every token to every token costs n². At 100,000 tokens that grid has ten billion cells.
Generation must remember every past key and value. Past a certain context the cache, not the weights, owns your GPU.
Attention is starved for memory bandwidth, not arithmetic. The maths can be perfect and the kernel still slow.
A genealogy · vermilion marks the mechanisms still being argued about
Scroll the diagram sideways →