Volume I · Silicon

The Accelerator Zoo

Every accelerator generation ships with a headline FLOPS number, and every headline number invites the same mistake: assuming it is the thing that decides whether a training run or a serving fleet is fast. Read the last decade of NVIDIA, Google, and AMD datasheets side by side and a different number keeps doing the actual work — memory bandwidth, the rate at which a chip can move data between its high-bandwidth memory and its compute cores.

NVIDIA's line: A100 to GB200

NVIDIA's A100, launched in 2020, is the baseline this whole zoo gets measured against. Its official datasheet lists 312 TFLOPS of dense FP16/BF16 Tensor Core throughput (624 TFLOPS with structured sparsity), paired with either a 40GB HBM2 configuration at 1,555 GB/s or an 80GB HBM2e configuration at roughly 2 TB/s of memory bandwidth. GPU-to-GPU, the third generation of NVLink moves 600 GB/s per GPU. Two years later, H100 roughly doubled the compute figure and then some — its datasheet claims 1,979 TFLOPS of dense BF16 and, through the Transformer Engine's FP8 path, 3,958 TFLOPS — while memory moved to 80GB of HBM3 at 3.35 TB/s and NVLink 4 lifted the per-GPU interconnect figure to 900 GB/s.

H200, released in 2024, is the generation that makes the lecture's whole point in one product. It sits on essentially the same compute die as H100 — the FLOPS figures do not meaningfully move — and its entire pitch is a memory upgrade: 141GB of HBM3e at roughly 4.8 TB/s, against H100's 80GB at 3.35 TB/s. NVIDIA is not claiming a faster chip for compute-bound work. It is claiming a faster chip for workloads that are bottlenecked on reading large amounts of memory quickly — specifically, serving requests with large KV caches, the subject of Lecture 6. Blackwell's B200 pushes the memory line further still: up to 192GB of HBM3e per GPU and up to 8 TB/s of bandwidth, with a fifth generation of NVLink moving 1.8 TB/s per GPU. The GB200 NVL72 rack couples 72 Blackwell GPUs through a fifth-generation NVSwitch fabric that NVIDIA's own developer blog describes as delivering roughly 130 TB/s of all-to-all GPU bandwidth within the rack. One number worth flagging for care rather than quoting flatly: a single, uncontested per-GPU FP8 PFLOPS figure for Blackwell is harder to pin down than the memory and interconnect numbers above — treat the NVIDIA developer blog's framing of the rack as a unit, not a single disputed per-chip peak-FLOPS figure, as the more defensible citation.

Google's TPUs: from a training pod to "the age of inference"

Google's TPU line tells a parallel story with its own vocabulary. TPU v4, described in the ISCA 2023 paper by Jouppi et al. (arXiv:2304.01433), organizes 4,096 chips into a pod wired together through Optical Circuit Switches — a fabric that can be dynamically reconfigured rather than being fixed at manufacture time. The paper reports 2.1x the performance of TPU v3 and a 2.7x improvement in performance per watt, and makes a point of stating that the optical switches and optics add less than 5% to system cost and less than 3% to system power — the reconfigurability is close to free. TPU v5e, documented in Google Cloud's official TPU docs, is a smaller and more inference-friendly chip: 197 TFLOPS of BF16 per chip, 16GB of HBM2 at 819 GB/s, and an inter-chip interconnect (ICI) of 400 GB/s bidirectional per chip, arranged in pods of up to 256 chips over a 2D torus.

Trillium (TPU v6) and then Ironwood (TPU v7) push the same memory-first logic the NVIDIA line already showed. Google's own blog post introducing Ironwood is explicitly titled "the first Google TPU for the age of inference" — a deliberate contrast with TPU v4's training-era design goal. Ironwood's disclosed figures: 4,614 TFLOPS of peak FP8 per chip, 192GB of HBM, 7.37 TB/s of HBM bandwidth, and 1.2 TB/s of bidirectional ICI, available in pod configurations of 256 or up to 9,216 chips — a full 9,216-chip pod totals what Google states as 42.5 Exaflops. Relative to Trillium, Google claims 6x the HBM capacity, 4.5x the HBM bandwidth, 1.5x the ICI bandwidth, and 2x the performance per watt; relative to TPU v2 (2018), the blog claims roughly 30x the performance per watt. Naming a chip "for the age of inference" is not marketing flourish here — it is the same fact as H200's memory-only refresh: inference is memory-bandwidth-hungry rather than FLOPS-hungry, because every decoded token means reading the entire KV cache back out of memory.

AMD's answer: MI300X and MI325X

AMD's CDNA 3-based MI300X, per its official product page, ships with 192GB of HBM3 and a peak memory bandwidth of 5.325 TB/s — more raw memory than any single NVIDIA GPU discussed above at the time of its release — alongside roughly 1.3 PFLOPS of dense BF16/FP16 and roughly 2.6 PFLOPS of FP8. MI325X keeps the same compute and upgrades memory again, to 256GB of HBM3e at 6 TB/s. The pattern by now should be unsurprising: AMD's own refresh cycle follows the identical memory-first logic as H100→H200 and Trillium→Ironwood.

The wiring, briefly — and the ladder ahead

NVLink and NVSwitch bandwidth numbers above are not separate trivia; they are the first rung of a bandwidth ladder that this series returns to directly in Lecture 2. A100's 600 GB/s of NVLink per GPU, H100's 900 GB/s, and GB200 NVL72's roughly 130 TB/s of all-to-all bandwidth across 72 GPUs are three points on the same curve — intra-node and intra-rack bandwidth climbing generation over generation, even as the much slower inter-node fabric (InfiniBand, or Ethernet in the RoCE/Spectrum-X style) stays the bottleneck for anything that has to leave the rack.