Volume XI · Reading the Reports

The Shape of a 2026 Flagship

Lay the architecture diagrams of the last eighteen months side by side on a table and cover the names. Different continents, different laboratories, different licences — and a silhouette so consistent that you can tell within a page which parts are the convention and which part is the argument.

The skeleton: ultra-sparse experts

Every frontier model discussed in this course is a mixture of experts, and the striking trend is not that they are sparse but how fast the active fraction has fallen. DeepSeek-V3 activated 37B parameters out of 671B — about five and a half per cent — and that was considered aggressive at the end of 2024. Qwen3.5 activates 17B of 397B. gpt-oss-120b activates 5.1B of 117B. Kimi K2 activated 32B out of 1.04T, spread across 384 experts.

Kimi K3 was announced at 2.8T total parameters with roughly 16 of 896 experts active per token — under two per cent unconfirmed, since no technical report had appeared at the time of writing. Take the exact figure lightly and the direction seriously: total capacity and per-token cost have been decoupled, and the laboratories are pushing the ratio between them as far as routing and training stability permit.

The contested axis

Sparsity of experts is settled convention. Attention is not, and that is the argument of Volume IV restated as a summary you can now read fluently. Three camps, all in production, all documented.

The first makes the score grid sparse and trainable: compute only a selected subset of query–key pairs and learn the selection. DeepSeek's DSA in V3.2 and MiniMax's MSA in M3 are of this family, and GLM-5 pairs DSA with the latent attention of Lecture 4. The second replaces most attention layers with a linear-cost recurrence and keeps a minority of full-attention layers to do what recurrence cannot: Qwen3.5's Gated DeltaNet hybrid at three to one, Nemotron 3's Mamba-2 hybrid, Kimi K3's KDA hybrid. The third keeps full attention and makes it cheap by restricting its reach — gpt-oss alternates dense and sliding-window layers and adds learned attention sinks.

All three are answers to the same bill from Lecture 2, and none of them has won. What should be clear by now is why the disagreement persists: the three camps make different bets about which of the assumptions in § 1.5 is safest to break, and the evidence from serving at scale does not yet separate them.

Precision fell to four bits

The quieter revolution is numerical. DeepSeek-V3 normalised FP8 mixed-precision pretraining in 2024 — not quantisation after the fact, but training the model in eight-bit floating point with the loss curve to show for it. Within two years the field went to four.

NVIDIA's Nemotron 3 pretrained in NVFP4. gpt-oss ships its expert weights in MXFP4, which is the reason a 120B-parameter model fits on a single 80GB accelerator — a fact with more practical consequence for who can run frontier-class models than any architectural choice in this course. Kimi's K2 Thinking used INT4 quantisation-aware training, folding the target precision into the training process rather than hoping the weights survive being squeezed afterwards.

After reinforcement learning

Architecture stopped being the differentiator some time in 2025. Post-training took its place, and post-training is itself now changing shape.

The arc begins with DeepSeek-R1, which showed reasoning behaviour emerging from reinforcement learning against rule-based rewards — and in the R1-Zero variant, with no supervised fine-tuning at all. Kimi K2 contributed the MuonClip optimiser and QK-clipping, which together remain the best public account of optimiser instability at trillion scale: 15.5 trillion training tokens without a loss spike is a claim that says more about engineering discipline than any benchmark. GLM-5 contributed asynchronous reinforcement-learning infrastructure that decouples generation from training, which is an infrastructure result masquerading as a modelling one — the bottleneck in RL at this scale is keeping accelerators busy while a policy writes.

And then the pipeline appears to be turning again. DeepSeek-V4, released in April 2026, reportedly replaces much of the reinforcement-learning pipeline with on-policy distillation from a fleet of specialist teacher models unconfirmed — the primary paper was not available at the time of writing, which makes this exactly the kind of claim Lecture 23 asks you to date and hold loosely.

One million tokens

Context lengths converged. A frontier release in 2026 offers something on the order of a million tokens, and the number has stopped being a differentiator because everyone has it.

The point worth carrying out of this course is that the convergence was not a matter of ambition. A million-token context is a promise about an n × n grid with 10¹² cells, and it became affordable rather than merely advertised because of the mechanisms in these sixteen lectures: shared and compressed caches so the memory does not explode (Volume II), sparsity and windows so most of the grid is never computed (Volumes III and IV), linear-cost recurrence so the remainder scales gently (Volume V), tiling and paging so what is computed is computed at hardware speed and stored without waste (Lectures 12 and 13), and the extension recipes of Lecture 14 so the positions are legible when it gets there. Every volume of this course is, read from far enough away, a line item in making that one number real.

What this course did not cover

It is only honest to be explicit about the size of the omission. Attention is one axis of a much larger machine, and four of the other axes matter at least as much.

Mixture-of-experts routing — how a token chooses its experts, how load is balanced, why routing collapses and what prevents it — was treated here as a given, and it deserves its own series; § 16.1 counted experts without once explaining how one is selected. Tokenisation sits underneath everything and was never mentioned, though it determines what a "token" in every formula actually is. The reinforcement-learning and preference-optimisation recipes were compressed into a single section of a single lecture. Multimodal fusion — how images, audio and video enter the same residual stream — was left out entirely, and it changes the attention story more than any mechanism in Volume IV. And inference-time scaling, the decision to spend more computation at answer time rather than training time, is arguably the defining development of 2025 and appears here only obliquely.

Which returns us to the thesis. The reason attention was worth a whole course, rather than the chapter it usually gets, is that it is the one part of the architecture the field is still openly arguing about. Expert sparsity is convention. Four-bit training is convergent engineering. The million-token context is settled. Attention is where three incompatible answers are shipping simultaneously in flagship models, where the published evidence genuinely does not yet decide between them, and where a reader who understands the mechanisms can form a real opinion rather than adopt one.

Hold that opinion the way Lecture 11 holds its subject: provisionally, with the evidence attached, and with the failures reported alongside the successes. Then go and read the primary sources. Every mechanism in these sixteen lectures came out of a paper you can open today, and the next one is being written now.