Appendix
Model by Model
Every lecture in this series draws its numbers from somewhere. This appendix is that
somewhere, collected into one table per laboratory: release date, parameter count,
attention mechanism, context length, training precision, and — the two columns most
reports leave out — what precision the weights were actually released in, and what a
server needs to run them. A blank cell is not an omission on this page's part; it is
the fact. The laboratory did not say. Lecture 23 argues that the shape of the gaps
is itself the most honest thing a report tells you — this table is where that argument
becomes checkable.
How to read a row
Total / Active is parameters at rest vs. parameters touched per token — the
gap between the two is Volume VII's subject. Attention abbreviates the
mechanism Volumes I–VI name in full (MHA, GQA, MLA, hybrid ratios, sparse variants).
Train precision is the number format pretraining actually ran in, when a report
says so. Released / inference is the format the public weights ship in, or the
format a report specifically claims for serving — a different question from training
precision, and the one most reports skip entirely. unconfirmed
marks a figure from secondary coverage rather than a primary paper, model card, or
official repository.
DeepSeek
| Model | Released | Total / Active | Attention | Context | Train precision | Released / inference | Source |
| V3 | Dec 2024 | 671B / 37B | MLA | 128K | FP8 mixed-precision | not disclosed | 2412.19437 |
| R1 | Jan 2025 | 671B / 37B (V3 backbone) | MLA | 128K | not disclosed | not disclosed | 2501.12948 |
| V3.2 / V3.2-Exp | Dec 2025 | 671B / 37B | MLA + DSA | 128K | not disclosed | not disclosed | 2512.02556 |
| V4 Pro / Flash | Apr 2026 | 1.6T / 49B · 284B / 13B | hybrid sparse (unconfirmed detail) | 1M | not disclosed | not disclosed | release note unconfirmed |
Qwen — Alibaba
| Model | Released | Total / Active | Attention | Context | Train precision | Released / inference | Source |
| Qwen3 (dense + MoE) | May 2025 | 0.6B–32B dense · up to 235B / 22B MoE | GQA | 128K (extended) | not disclosed | not disclosed | 2505.09388 |
| Qwen3-Next | Sep 2025 | 80B / 3B | Gated DeltaNet : full, 3:1 | not disclosed | not disclosed | not disclosed | blog only |
| Qwen3.5-397B-A17B | Feb 2026 | 397B / 17B | Gated DeltaNet : gated full, 3:1 | 262K | not disclosed | not disclosed | model card — no technical report as of Feb 2026 |
Kimi — Moonshot AI
| Model | Released | Total / Active | Attention | Context | Train precision | Released / inference | Source |
| K2 | Jul 2025 | 1.04T / 32B (384 experts) | MLA | 128K | not disclosed | not disclosed | 2507.20534 |
| K2 Thinking | Nov 2025 | 1.04T / 32B | MLA | 128K | not disclosed | INT4, quantization-aware training | model card |
| K2.5 | Jan 2026 | ~1T / ~32B | MLA | not disclosed | not disclosed | not disclosed | model page |
| K3 | 16 Jul 2026 | 2.8T / ~16 of 896 experts (≈1.8%) | Kimi Delta Attention (linear hybrid) + MLA | 1M | not disclosed | not disclosed | technical report not yet published unconfirmed |
Meta — Llama
| Model | Released | Total / Active | Attention | Context | Train precision | Released / inference | Source |
| Llama 4 Scout | Apr 2025 | 109B / 17B (16 experts) | iRoPE (RoPE local + NoPE global) | 10M | FP8 | not disclosed | blog only, no paper |
| Llama 4 Maverick | Apr 2025 | 400B / 17B (128 experts) | iRoPE | not disclosed | FP8 | not disclosed | blog only |
| Llama 4 Behemoth | never shipped | ~2T / 288B (previewed) | not disclosed | not disclosed | not disclosed | not disclosed | preview only |
Google — Gemini & Gemma
| Model | Released | Total / Active | Attention | Context | Train precision | Released / inference | Source |
| Gemini 2.5 Pro / Flash | 2025 | sparse MoE, size not disclosed | not disclosed | 1M | not disclosed | not disclosed | 2507.06261 |
| Gemini 3 Pro | Nov 2025 | sparse MoE, size not disclosed | not disclosed | 1M in / 64K out | not disclosed | not disclosed | model card only |
| Gemini 3.1 Pro | Feb 2026 | sparse MoE, size not disclosed | not disclosed | 1M | not disclosed | not disclosed | model card only |
| Gemma 4 | Jun 2026 (report) | 31B dense · 26B / 4B MoE | not disclosed | 256K (31B) | not disclosed | Apache 2.0 weights, format not disclosed | 2607.02770 |
OpenAI
| Model | Released | Total / Active | Attention | Context | Train precision | Released / inference | Source |
| GPT-5 | Aug 2025 | not disclosed (router of models) | not disclosed | not disclosed | not disclosed | not disclosed | system card only |
| gpt-oss-120b | Aug 2025 | 117B / 5.1B | alternating dense + sliding window, learned sinks | 131K (YaRN-extended) | not disclosed | MXFP4 (expert weights) — fits one 80GB GPU | 2508.10925 |
| gpt-oss-20b | Aug 2025 | 21B / 3.6B | alternating dense + sliding window, learned sinks | 131K | not disclosed | MXFP4 (expert weights) | 2508.10925 |
| GPT-5.1 / 5.5 / 5.6 (Luna, Terra, Sol) | Nov 2025 – Jul 2026 | not disclosed | not disclosed | not disclosed | not disclosed | not disclosed | system cards / preview posts only |
Anthropic
| Model | Released | Total / Active | Attention | Context | Train precision | Released / inference | Source |
| Opus 4.1 / Sonnet 4.5 / Opus 4.5 | Aug–Nov 2025 | not disclosed | not disclosed | not disclosed | not disclosed | not disclosed | system cards only |
| Opus 4.8 | May 2026 | not disclosed | not disclosed | not disclosed | not disclosed | not disclosed | system card only |
| Fable 5 / Mythos 5 | Jun 2026 | not disclosed | not disclosed | 1M in / 128K out (third-party reported) | not disclosed | not disclosed | system card only |
Zero architecture rows in this table have a single disclosed cell — that emptiness,
across an entire laboratory, is itself the data point Lecture 23 asks you to notice.
Mistral
| Model | Released | Total / Active | Attention | Context | Train precision | Released / inference | Source |
| Magistral | Jun 2025 | (Mistral Medium 3 backbone) | not disclosed | not disclosed | not disclosed | not disclosed | 2506.10910 |
| Mistral Large 3 | Dec 2025 | 675B / 41B | not disclosed | 256K | not disclosed | Apache 2.0 weights, format not disclosed | blog + weights, no paper |
GLM — Zhipu / Z.ai
| Model | Released | Total / Active | Attention | Context | Train precision | Released / inference | Source |
| GLM-4.5 (+ Air) | Jul 2025 | 355B / 32B · Air 106B / 12B | hybrid thinking modes | not disclosed | not disclosed | not disclosed | 2508.06471 |
| GLM-5 | Feb 2026 | 744B / 44B | MLA + DSA (adopted from DeepSeek) | 200K | not disclosed | not disclosed | 2602.15763 |
MiniMax
| Model | Released | Total / Active | Attention | Context | Train precision | Released / inference | Source |
| M1 | Jun 2025 | 456B / 45.9B | Lightning (linear) hybrid | 1M | not disclosed | not disclosed | 2506.13585 |
| M2 | model Oct 2025, report May 2026 | 230B / 10B (256 experts) | full attention (deliberate reversion) | 192K | not disclosed | not disclosed | 2605.26494 |
| M3 | Jun 2026 | 428B / 22B | GQA + MiniMax Sparse Attention | 1M | not disclosed | not disclosed | 2606.13392 |
xAI — Grok
| Model | Released | Total / Active | Attention | Context | Train precision | Released / inference | Source |
| Grok 4 | Jul 2025 | ~3T total (reported) | not disclosed | not disclosed | not disclosed | not disclosed | model card only |
| Grok 4.1 / 4 Fast | Nov 2025 | not disclosed | not disclosed | 2M (Fast) | not disclosed | not disclosed | model cards only |
| Grok 4.5 | Jul 2026 | ~1.5T "V9 foundation" (reported) | not disclosed | not disclosed | not disclosed | not disclosed | single social post unconfirmed |
NVIDIA — Nemotron
| Model | Released | Total / Active | Attention | Context | Train precision | Released / inference | Source |
| Nemotron 3 Nano | Dec 2025 | 30B / 3B | Mamba-2 hybrid | 1M-class | NVFP4 | NVFP4 | 2512.20856 |
| Nemotron 3 Super | Dec 2025 | ~120B / 12B | Mamba-2 hybrid | 1M-class | NVFP4 | NVFP4 | 2512.20856 |
| Nemotron 3 Ultra | Dec 2025 | 550B / 55B | Mamba-2 hybrid | 1M-class | NVFP4 — first 4-bit pretrain at this scale | NVFP4 | 2512.20856 |
Ai2 — OLMo
| Model | Released | Total / Active | Attention | Context | Train precision | Released / inference | Source |
| OLMo 3 (7B, 32B) | Nov 2025 | 7B / 7B · 32B / 32B (dense) | not disclosed | not disclosed | disclosed in full training code (not summarised here) | full weights + every checkpoint, native format | blog + full repo — the only fully reproducible release in this table |
Other notable releases
| Model | Released | Total / Active | Attention | Context | Train precision | Released / inference | Source |
| Falcon-H1 (TII) | 2025 | not disclosed | parallel Mamba-2 + attention, same block | not disclosed | not disclosed | not disclosed | 2507.22448 |
| Jamba 1.7 (AI21) | 2025 | not disclosed | Mamba + attention + MoE hybrid | 256K | not disclosed | not disclosed | original Jamba: 2403.19887 |
| Command A (Cohere) | Mar 2025 | 111B dense | not disclosed | 256K | not disclosed | not disclosed | 2504.00698 |
| SmolLM3 (Hugging Face) | Jul 2025 | 3B dense | NoPE on every 4th layer | 128K (staged 4K→64K→128K) | disclosed in full training code | fully open weights + data + code | blog |
| Hunyuan 3.0 "Hy3" (Tencent) | Jul 2026 | 295B / ~21B (192 experts) | not disclosed | 256K | not disclosed | FP8 release | coverage only unconfirmed |
| ERNIE 5.0 (Baidu) | Feb 2026 | not disclosed | not disclosed | not disclosed | not disclosed | not disclosed | 2602.04705 |
What the blanks add up to
Count the disclosed "Released / inference" cells above: four labs state a released or
serving precision in a primary source — OpenAI (MXFP4, gpt-oss only, its one
forced-open release), NVIDIA (NVFP4, the whole Nemotron 3 line), Kimi
(INT4 QAT, the Thinking variant only), and Ai2 (native, because OLMo 3 ships
everything). Every frontier closed lab — OpenAI's flagship line, Anthropic, Google's
Gemini, xAI — discloses nothing about the number format its weights actually run in.
That is not this page failing to find the fact. It is the fact.
Where these rows come from
- Primary sources are linked per row above; anything without a direct link was verified against the model's official blog, card, or repository rather than a paper.
- Volume VII (lectures 15–16) explains the architecture column in full; Volume VIII (17–18) explains the two precision columns; Volume XI (23–24) is the argument this table exists to support.
- Sebastian Raschka's LLM Architecture Gallery — the closest existing equivalent to this table, and worth cross-checking against.