Training Infrastructure

Compute is now measured in gigawatts.

Frontier compute is planned in gigawatts and multi-year contracts, not GPU counts. In 2026, NVIDIA Blackwell Ultra (GB300) is the shipping workhorse and Vera Rubin systems arrive in the second half of the year; Google's TPU v7 Ironwood and AWS Trainium3 are credible non-NVIDIA paths, and Anthropic trains across all three platforms. OpenAI's GPT-6 Astra, its largest run "by far", reportedly used more than 100k GPUs at Stargate Abilene; xAI's Colossus 2 is described as the first gigawatt-scale training cluster. Inside the cluster the recipe is 5D parallelism (data × tensor × pipeline × expert × context), FP8 as routine with NVFP4 emerging, the Muon optimizer at trillion-parameter scale, asynchronous RL infrastructure, and relentless fault tolerance.

Why it matters

Epoch's long-run trend is about 5× per year growth in frontier training compute. Power and grid interconnects, not chips, are now the binding constraint on new sites.
OpenAI says it passed its 10 GW contracted US target in April 2026; its NVIDIA, AMD and Broadcom deals total about 26 GW, deploying from H2 2026 through 2029 (Epoch).
Anthropic runs on Google TPUs (up to 1M chips and over 1 GW in 2026, per press reports), AWS Trainium (Project Rainier, about 500k Trainium2 chips) and NVIDIA. Multi-platform portability is now a strategic capability.
Low precision compounds: FP4 GEMMs run about 2–3× faster than FP8 on GB200/GB300, and NVIDIA showed 4-bit pretraining of a 12B model on 10T tokens matching FP8 (MMLU-Pro 62.58% vs 62.62%).
Hardware failures scale with cluster size. Meta's Llama 3 run logged 419 interruptions in 54 days on 16k GPUs; at 100k+ accelerators, fault tolerance decides your effective MFU.

State of the art

2025-2026

NVIDIA roadmap on a yearly cadence: Blackwell Ultra (GB300 NVL72) shipping; Vera Rubin (NVL72/NVL144) in H2 2026; Rubin Ultra (Kyber rack, HBM4e) in 2027; Feynman in 2028.
Google TPU v7 Ironwood: 4.6 PFLOPs FP8 per chip, 192 GB HBM3e, 7.37 TB/s; GA late 2025 with mass deployment in 2026. Gemini 4 Argon was trained on TPUs with JAX and Pathways. Google's 8th generation is reportedly split into separate training (Broadcom) and inference (MediaTek) chips on TSMC 2nm for late 2027.
AWS Trainium3: 3nm, 2.52 PFLOPs FP8, 144 GB HBM3e, 4.9 TB/s per chip; a Trn3 UltraServer packs 144 chips for 362 PFLOPs (GA announced Dec 2025).
AMD and custom silicon: AMD (MI355X and successors) and Broadcom custom accelerators are part of OpenAI's multi-vendor 26 GW pipeline.
OpenAI Stargate Abilene: 6 of 8 buildings (618 MW) delivered by September 2026; press reports say a planned 2 GW expansion there was cancelled.
xAI (now part of SpaceX): Colossus 2 came online in January 2026 and is described as the first GW-class training cluster. NVIDIA committed $5B to Safe Superintelligence (Jul 2026) with access to Vera Rubin.
Training recipes disclosed in open reports: DeepSeek-V4 (1.6T total / 49B active, 32T+ tokens) and Kimi K3 (2.8T / 104B) use Muon; FP8 is routine; asynchronous RL systems (slime/Miles, GLM-5) decouple rollout generation from training to keep accelerators busy.

The recipe

A frontier-grade implementation, in order.

Secure power and capacity first

Plan 1–2 years ahead for megawatts, cooling (liquid for GB300 and Rubin racks) and interconnect. Diversify across vendors (NVIDIA, TPU, Trainium, AMD) if your software stack can absorb it; contracts now run multi-gigawatt and multi-year.

Cluster spec

Rack-scale NVLink domains (NVL72 and up) or TPU/Trainium pods for tensor and expert parallelism; InfiniBand or high-radix Ethernet for scale-out; a parallel filesystem plus NVMe scratch sized for multi-terabyte checkpoints.

Parallelism strategy

DP × TP × PP × EP × CP. Keep TP and EP inside a rack-scale NVLink domain, PP across racks, context parallelism (ring attention) for 1M-token training, and FSDP/ZeRO-3 sharding for parameters, gradients and optimizer state. Megatron-LM and JAX/Pathways are the reference stacks.

Precision and optimizer

FP8 matmuls with higher-precision master weights as the default; trial NVFP4 with random Hadamard transforms, 2D block scaling, stochastic rounding and a few high-precision layers. Muon's Newton-Schulz orthogonalization needs distributed implementations over sharded weights; budget its communication.

Overlap communication

Overlap collectives with compute everywhere. For MoE with 3–4% active parameters, the all-to-all is the bottleneck; use DualPipe-style bidirectional pipelines, in-network reduction and expert placement aware of topology.

RL infrastructure

Separate rollout fleets (inference engines such as SGLang or vLLM) from trainer fleets, sync weights asynchronously, and run sandboxed agent environments at scale. RL now consumes a large share of frontier compute; treat it as a first-class system.

Fault tolerance and telemetry

Asynchronous checkpointing every 15–30 minutes, silent-data-corruption detection, automatic node fencing and hot spares. Track per-layer gradient and activation norms, loss spikes and per-node throughput; aim for MFU of 40% or more.

⚠️

Common pitfalls

Announced gigawatts are not delivered gigawatts. Track buildings energized and chips installed; site plans change (for example, the reported cancellation of the Abilene 2 GW expansion).

Silent data corruption grows with cluster size. Without per-step gradient sanity checks, you can train for days on bad math.

MoE all-to-all collectives decide MFU. Poor expert placement or no compute/communication overlap can cost 20 points of utilization.

Low-precision training fails late: some layers can drift in FP4, and Muon's Newton-Schulz iteration has a reported failure regime in some late layers at frontier scale. Keep fallbacks to higher precision and monitor spectral statistics.

Multi-vendor fleets multiply software cost. Kernels, collectives and numerics differ across GPUs, TPUs and Trainium; budget engineering for parity testing.

Storage and checkpoint I/O often bottleneck before compute. Provision parallel filesystem bandwidth for multi-terabyte checkpoints.