Volume IV · The Reality of Scale

The Utilization Nobody Reaches 100% Of

Lecture 1 quoted a chip's peak FLOPS off a spec sheet, and every lecture since has quietly assumed a training run gets close to using it. It does not. This lecture is the volume's reality check — the gap between advertised hardware compute and the compute a real, days-long training job actually turns into forward-and-backward passes, and the interruptions that gap is partly made of.

Defining MFU, in the paper that named it

"Model FLOPs Utilization" (MFU) is commonly cited as having been named, as a distinct metric from raw hardware FLOPs utilization, in Chowdhery et al., "PaLM: Scaling Language Modeling with Pathways" (arXiv:2204.02311, Google, 2022). The distinction the paper draws matters: MFU counts only the FLOPs that do useful work toward the model's forward and backward pass — not the extra FLOPs spent recomputing activations that were deliberately discarded to save memory (rematerialization), and not FLOPs spent idle waiting on communication. MFU is defined as (actual useful FLOPs achieved) / (theoretical peak FLOPs of the hardware). A separate, looser number — hardware FLOPs utilization — counts rematerialization FLOPs as "achieved" too, so it is always at least as high as MFU for the same run.

PaLM 540B's own disclosed numbers make the gap concrete: 46.2% MFU counting attention FLOPs (45.7% excluding them), against 57.8% hardware FLOPs utilization including rematerialization — training throughput was 238.3K tokens/sec at batch size 2048. Read that pair of numbers as a waterfall: start from peak hardware FLOPS, subtract everything spent on rematerialization and idle communication to land at hardware FLOPs utilization, then subtract the rematerialization FLOPs themselves to land at MFU — the number that actually describes useful work.

Why it's an average that hides an interruption record

A single MFU number, quoted for a whole training run, is already an average over however many days the run took — and large synchronous training jobs are fault- intolerant at the job level: one GPU failure during a synchronous step can force a restart of the whole job, not just a graceful degradation of one worker. So the interruption rate, not the interruption count, is the operative reliability metric — a cluster that fails often but recovers in seconds behaves very differently from one that fails rarely but needs hours to restart from checkpoint.

Meta's Llama 3 paper (arXiv:2407.21783, 2024) discloses exactly this kind of record for a 54-day pretraining snapshot on its 16,384-H100 cluster: 466 total job interruptions, of which 47 were planned or automated maintenance and 419 were unexpected. Of those unexpected interruptions, roughly 78% were attributed to confirmed hardware issues. The largest single category was GPU issues, including NVLink failures, at 148 of 419 (about 30.1%); HBM3 memory failures were the second largest at 72 of 419 (about 17.2%). Despite that volume of interruptions, the paper reports over 90% effective training time — automation handled nearly all of the interruptions, and only 3 of the 466 required manual intervention.

Read this alongside the MFU waterfall above: a large fraction of the distance between peak hardware FLOPS and a run's realized MFU is not an inefficiency in the matrix multiply itself — it is checkpoint-and-restart time paid every time a GPU, an NVLink link, or a memory module fails mid-run, at a scale where hundreds of such failures over less than two months is the disclosed norm, not an anomaly.

DeepSeek-V3's cost figure, and a number DeepSeek did not publish

DeepSeek-V3 (arXiv:2412.19437, 2024) is a 671B-total-parameter MoE model with 37B active parameters per token, trained on 14.8T tokens. The paper's own disclosed figure for full pretraining cost is 2.788M H800 GPU-hours — about 180K H800 GPU-hours per trillion tokens — and the paper's own cost estimate assumes $2/GPU-hour rental.

A logbook, not a paper — and why that's a different kind of source

The OPT-175B "Chronicles" logbook (Meta AI, published in the facebookresearch/metaseq GitHub repository as OPT175B_Logbook.pdf) is an unusually candid primary source precisely because it is not a paper — Meta published the raw on-call engineering logbook itself. Secondary summaries of that logbook describe roughly 35 training restarts and 100+ hosts cycled during training, with hardware failures — "a couple of machines going down every day" — described as routine on the roughly 1024-A100 deployment. Those restart and failure counts are worth stating explicitly as pulled from secondary summaries of the logbook rather than independently re-verified against the primary PDF line by line — the honest thing to do before citing an exact number from it elsewhere is to open OPT175B_Logbook.pdf directly rather than trust a summary of a summary.

The throughline across all four sources is the same: peak hardware FLOPS is a marketing number, hardware FLOPs utilization is a generous accounting number, and MFU is the honest one — and even the honest one, at the flagship-lab level, sits under 50%, with a meaningful share of the shortfall traceable to hardware failing often enough that "how fast does the job recover" is as load-bearing a design question as the matrix-multiply kernel itself.