Pretraining at Scale

The run where everything has to go right — now in FP8, trending to FP4, with Muon.

A frontier pretraining run in late 2026 occupies 100k+ Blackwell-class GPUs. GPT-6 Astra (Sep 2026) was reportedly trained on more than 100k GPUs at Stargate Abilene, and OpenAI says only that it was 'by far' its largest run. Open recipes disclose 28-32T+ tokens. Three things changed since 2025. First, Muon (orthogonalized updates via Newton-Schulz) replaced AdamW as the main optimizer in every disclosed open frontier recipe: Kimi K2, GLM-5, DeepSeek-V4 and Kimi K3. Second, FP8 is routine and NVFP4 4-bit pretraining has been shown to match FP8 at 12B / 10T tokens. Third, multi-token prediction and staged long-context extension to 1M tokens are table stakes. Scaling-law sweeps, μP-style hyperparameter transfer and a high-quality annealing phase are still how you de-risk the run.

Why it matters

Frontier training compute grows ~5x per year on Epoch's long-run trend. A single failed run wastes months of the world's largest clusters, so most labs run many small ablations for every full run.
The optimizer is no longer settled. Moonshot's 'Muon is Scalable for LLM Training' (Feb 2025) started the shift, and by 2026 Muon is the disclosed main optimizer in DeepSeek-V4, Kimi K3 and GLM-5.
Precision is the cheapest speedup left. FP4 GEMMs run ~2-3x faster than FP8 on GB200/GB300, and NVIDIA's NVFP4 run (12B hybrid Mamba-Transformer, 10T tokens) scored 62.58% MMLU-Pro vs 62.62% for FP8.
Inference-aware over-training continues. Small models see thousands of tokens per parameter (Ouro: 7.7T tokens at 1-3B), because serving cost dominates lifetime cost.

State of the art

2025-2026

Muon is the default optimizer for open frontier MoEs (Kimi K2, GLM-5, DeepSeek-V4, Kimi K3). The ICLR 2026 Honorable Mention 'Polar Express' gives optimal matrix-sign iterations for it, and 'Spectral Scaling Laws of Muon' (2026) warns that some late layers fall into a Newton-Schulz failure regime at frontier scale.
FP8 pretraining (proven by DeepSeek-V3) is routine. NVFP4 uses random Hadamard transforms, 2D block scaling, stochastic rounding and a few high-precision layers. TetraJet-v2 and UE5M3 FP4 block scaling (Sep 2026) push 4-bit further.
Scale in open reports: DeepSeek-V4-Pro (Apr 2026) is 1.6T total / 49B active on 32T+ tokens with 1M context. GLM-5 (Feb 2026) is 744B/40B on 28.5T. Kimi K3 (Jul 2026) is the largest open-weight model at 2.8T/104B.
Multi-token prediction is now standard (DeepSeek-V3/V4, Nemotron 3, Qwen3-Next). It adds a training signal and a self-speculation head.
Warmup-Stable-Decay schedules, best-fit packing with cross-document masking, and a mid-training stage (high-quality math/code/reasoning plus long documents) are now standard hygiene.
Closed labs disclose little. Press reports put GPT-6 Astra on 100k+ GPUs at Stargate Abilene, while OpenAI says only that it was 'by far' its largest run. FLOP estimates circulating for it are third-party guesses, not lab figures.

The recipe

A frontier-grade implementation, in order.

Scaling-law sweep

Train 8-12 small models (10M-1B params, ideally with your final architecture: hybrid attention, MoE sparsity, MTP) at varying token counts. Fit constants on YOUR data. Fit optimizer-specific laws too, because Muon and AdamW optima differ.

Hyperparameter transfer

Tune LR, init scale, batch size and Muon's Newton-Schulz settings on a small proxy and transfer with μP-style scaling. Re-check per-layer spectral behaviour at intermediate scale, since that is where Muon failure modes first appear.

Token budget

For a model you will serve at scale, go well past Chinchilla-optimal (Sardana inference-aware). The 2026 open frontier sits at ~28-32T+ tokens for 1-3T-total MoEs.

Optimizer

Muon for 2-D weight matrices, AdamW for embeddings, norms and output head (the standard split). Monitor update RMS and spectral norms per layer.

Precision

FP8 GEMMs with BF16/FP32 master weights and optimizer states as the safe default. On Blackwell, ablate NVFP4 with Hadamard transforms, 2D block scaling and stochastic rounding, keeping sensitive layers (embeddings, output, last few blocks) in higher precision.

Curriculum & context

Stage 1: broad mix at 4-8k context. Stage 2: mid-training anneal on high-quality math/code/reasoning. Stage 3: staged long-context extension to 128k → 1M on long documents. Keep MTP heads (loss weight ~0.1) on throughout.

⚠️

Common pitfalls

Loss spikes at scale: keep gradient, activation and update-norm telemetry at every layer. Restart from the last clean checkpoint and skip the offending batch.

Muon is not free: Newton-Schulz iterations can fail on late layers at frontier scale, and the extra matmuls need careful distribution across expert-parallel shards.

Four-bit training is unforgiving. Without Hadamard rotations and stochastic rounding, FP4 runs drift. Validate against an FP8 twin for the first few hundred billion tokens.

Don't change the data mix mid-run unless you've ablated it. The 'just one more high-quality dataset' temptation has killed many runs.

Long-context extension can quietly degrade short-context quality. Track both throughout stage 3.