Volume I · Silicon

Wiring the Datacenter Together

A rack full of the fastest chips in Lecture 1 is still only as fast as the wire connecting them. A real cluster is not one fabric — it is at least two, layered: an extremely fast one inside a node or a rack, and a much slower one stitching racks together into tens of thousands of GPUs. Which parallelism strategy you can afford is a direct consequence of which layer it has to cross, a point this lecture sets up and Lectures 3–5 build entire strategies around.

Two different jobs: intra-node and inter-node fabric

NVLink and NVSwitch, introduced already in Lecture 1 as bandwidth numbers, are doing a specific job: connecting GPUs that sit close together — inside one server, or across the tightly-coupled rack a system like GB200 NVL72 forms. NVIDIA's developer blog describes the NVL72 rack's fifth-generation NVSwitch fabric as delivering 14.4 TB/s of aggregate bidirectional bandwidth per NVSwitch chip — 72 ports at 200 GB/s each — and states the resulting all-to-all bandwidth across the full 72-GPU NVLink domain at roughly 130 TB/s. That is the fastest tier available in a modern cluster, and it only reaches as far as the rack's own copper and optical backplane.

Once a workload needs GPUs in a different rack, it is riding a different fabric entirely: InfiniBand, or Ethernet run in a lossless, RDMA-capable mode (RoCE — RDMA over Converged Ethernet). Both exist to move data between server nodes across a datacenter-scale network, and both operate at a per-link bandwidth that is a small fraction of what NVSwitch delivers inside a rack. This is the second, and much slower, tier — and it is the one every long-range parallelism strategy has to reckon with.

Meta's Llama 3 cluster: the disclosed worked example

Meta's own engineering blog, "Building Meta's GenAI Infrastructure" (March 2024), is one of the few primary-source, vendor-authored disclosures of what a real frontier-scale training cluster looks like end to end. It describes two separate 24,576-GPU H100 clusters: one built on RoCE (using Arista 7800-series Ethernet switches), the other on NVIDIA Quantum-2 InfiniBand — both running at 400 Gbps per NIC. Both clusters run on Meta's own "Grand Teton" GPU hardware platform, which Meta has contributed to the Open Compute Project. Building both an Ethernet-based and an InfiniBand-based cluster of the same size was itself a deliberate comparison — Meta wanted a real operational answer to whether a RoCE-based fabric could match InfiniBand at this scale, rather than taking a vendor's word for it.

One distinction is worth being precise about, because it is an easy one to blur: a cluster's total size is not the same fact as how many GPUs one training run actually used. The 24,576-GPU figure describes the physical clusters Meta built. The Llama 3 paper itself (arXiv:2407.21783) states that the 405B-parameter model was trained on up to 16,384 H100 GPUs — a real, smaller number, drawn from within that larger built capacity. Citing the cluster's size as the training run's size is a common and avoidable error.

DGX, HGX, SuperPOD: NVIDIA's reference shapes

NVIDIA sells the building blocks these clusters are assembled from as named reference architectures rather than a single product. DGX is NVIDIA's own single-node reference system — an 8-GPU server with NVLink/NVSwitch wiring already designed in. HGX is the corresponding reference board design that other server vendors license and build their own systems around, using the same GPU-to-GPU topology. SuperPOD is the reference architecture one level up: a full multi-node cluster built from many DGX or HGX nodes, combining NVLink/NVSwitch for the intra-node tier with InfiniBand or Ethernet for the inter-node tier — the same two-tier structure this lecture has been describing, packaged as a design NVIDIA publishes rather than one every operator has to invent from scratch.

Colossus: read this section as press reporting, not disclosure

xAI's "Colossus" cluster is worth covering because it is widely cited — and worth flagging clearly, because almost everything reported about it comes from a Supermicro customer case study and trade-press coverage, not from an xAI-authored engineering disclosure comparable to Meta's blog post above. Treat every figure in this section as industry reporting, not verified fact. With that stated plainly: press coverage reports roughly 100,000 H100 GPUs brought online in about 122 days at a Memphis, Tennessee site; a claim that NVIDIA's Spectrum-X Ethernet fabric achieved roughly 95% effective throughput against a traditional Ethernet fabric's roughly 60%; and a reported power draw around 250 MW. None of these numbers carries the evidentiary weight of a vendor's own engineering write-up, and a student should hold them loosely — as "this is what was reported," not "this is what was measured and disclosed."

The three-tier hierarchy, together

Put Lecture 1's chip numbers and this lecture's cluster numbers on one ladder and the shape becomes obvious: NVLink inside a single server, faster; NVSwitch across a rack (GB200 NVL72's ~130 TB/s all-to-all), faster still in aggregate but bounded to 72 GPUs; and InfiniBand or RoCE Ethernet between racks, the slowest tier and the one with no real upper bound on how many GPUs it can eventually reach. Every parallelism strategy in the next three lectures is, at bottom, a decision about which of these three tiers a particular kind of communication is allowed to cross.