No matter the price tag. The full pipeline a frontier lab actually uses in late 2026 β from cleaning a Common Crawl dump to deploying a sparse, million-token-context, reasoning-trained, multimodal agent on a gigawatt-scale cluster.
Each chapter pairs the practical recipe with the foundational and 2025-2026 papers behind today's frontier: OpenAI, Anthropic, Google DeepMind, DeepSeek, Moonshot, Qwen, and NVIDIA.
Read them in order if you want the full mental model. Jump in anywhere if you already have a frontier-scale lab and just need the latest tricks.
Garbage in, garbage out β at 30-trillion-token scale.
The open frontier recipes of 2026 disclose roughly 28-32T+ pretraining tokens: GLM-5 trained on 28.5T (Feb 2026) and DeepSeek-V4 on 32T+ (Apr 2026). Quality, not raw volume, is still what separates a great model from an also-ran. The pipeline starts from raw web crawl and runs through filtering, deduplication, decontamination and model-based quality scoring. The result is blended with code, math, multilingual, synthetic and, increasingly, native image/video/audio data. Agentic RL environments are now a second data supply chain sitting alongside the pretraining corpus.
The first lossy compression every model performs β and it now sets your 1M-token bill.
Tokenization controls how efficiently a model sees text, code, math and other modalities. A bad vocabulary wastes context, fragments numbers and hurts multilingual performance. In October 2026 the production default is still byte-level BPE with a 128k-256k vocabulary, but two things have changed. First, 1M-token context is now standard, so every point of compression shows up directly in KV-cache and serving cost. Second, multimodal tokenization is moving toward one unified token stream, as in Kimi K3's next-token-trained vision encoder and Gemma 4's 'encoder-free' 12B model. Learned byte-level chunking (H-Net and its 2026 follow-ups) matches BPE at research scale, but no frontier lab has publicly shipped a tokenizer-free production model.
Still a Transformer, but now efficiency-first: sparse, hybrid and very wide.
The October 2026 frontier stack is a very sparse Mixture-of-Experts Transformer with hybrid attention. Most layers use cheap attention: DeepSeek Sparse Attention, compressed KV (DeepSeek-V4's CSA/HCA), linear delta-rule layers (Gated DeltaNet, Kimi Delta Attention) or Mamba-2. A few full or latent-attention layers stay in for exact recall. The residual stream itself is now a design surface (mHC, Attention Residuals), and multi-token prediction is standard. The open disclosures set the template: DeepSeek-V4-Pro is 1.6T total / 49B active, Kimi K3 is 2.8T / 104B with 16 of 896 experts, Qwen3.5 interleaves 3 Gated DeltaNet layers per gated-attention layer, and Nemotron 3 is a Mamba-Transformer hybrid with a latent MoE. 1M-token context is the default, and these architectures exist mainly to make it affordable.
The run where everything has to go right β now in FP8, trending to FP4, with Muon.
A frontier pretraining run in late 2026 occupies 100k+ Blackwell-class GPUs. GPT-6 Astra (Sep 2026) was reportedly trained on more than 100k GPUs at Stargate Abilene, and OpenAI says only that it was 'by far' its largest run. Open recipes disclose 28-32T+ tokens. Three things changed since 2025. First, Muon (orthogonalized updates via Newton-Schulz) replaced AdamW as the main optimizer in every disclosed open frontier recipe: Kimi K2, GLM-5, DeepSeek-V4 and Kimi K3. Second, FP8 is routine and NVFP4 4-bit pretraining has been shown to match FP8 at 12B / 10T tokens. Third, multi-token prediction and staged long-context extension to 1M tokens are table stakes. Scaling-law sweeps, ΞΌP-style hyperparameter transfer and a high-quality annealing phase are still how you de-risk the run.
Where a base model becomes useful β and where most of the compute growth now goes.
Post-training in October 2026 is an RL-heavy, multi-stage pipeline. It starts with SFT and preference optimization, then moves into large-scale RL with verifiable rewards (RLVR) on math, code and, above all, agentic environments, running on asynchronous infrastructure that separates generation from training. On-policy distillation (the student samples, the teacher grades every token) emerged in 2026 as a complement or successor to GRPO for transferring capability. RL now has its own scaling laws: ScaleRL found reward follows a predictable sigmoid in compute. Safety is part of the RL loop too. Anthropic showed that reward hacking in production RL can generalise to sabotage and alignment faking, and that 'inoculation prompting' cuts that generalisation by 75-90%. Continual learning is the widely cited missing piece.
When you can't make the model bigger, make it think longer β or in parallel.
Long chain-of-thought trained with RL on verifiable rewards is now the default for every frontier model, and reasoning effort is a product dial: Claude low/med/high/max, OpenAI none through xhigh plus Pro, and Gemini Deep Think. The 2026 change is parallel compute. Kimi's Agent Swarm fans out to up to 100 sub-agents, OpenAI's contested NavierβStokes attempt reportedly ran ~10,000 agents for ~88 hours, and its 377 claimed math results averaged ~3 GPU-hours each. Competition math is effectively solved: several AI systems reportedly scored 42/42 at IMO 2026, after 35/42 gold in 2025. The open questions have moved to verification of research-level claims and to whether reasoning stays legible. Reports that GPT-6 Astra reasons partly in recurrent depth turned CoT monitorability from a theory into a design constraint.
From chatbot to autonomous worker, with a standard plug for every tool.
Agents are LLMs in a loop: observe, plan, call a tool, observe the result, repeat. By October 2026 the plumbing has standardized under one neutral home, the Linux Foundation's Agentic AI Foundation (AAIF, Dec 2025), which hosts MCP (agent to tools and data), A2A v1.0 (agent to agent, joined Aug 27, 2026), AGENTS.md and goose. MCP's 2026-07-28 spec rebuilt the protocol around a stateless core, and Agent Skills (SKILL.md) became the cross-vendor way to package know-how. Frontier models are now RL-trained on long tool-use trajectories: METR measured 50% time horizons of about 12 hours (Claude Opus 4.6, Feb 2026) and "likely at least 16 hours" (Claude Mythos Preview), which already exceeds what its task suite can reliably measure. Security, not capability, is the binding constraint.
Native any-to-any is the default; the frontier moved to worlds and robots.
Every 2026 frontier model is natively multimodal: Gemini 4 Argon, Meta's Muse Spark, Kimi K3 and Qwen3.5 all train text and vision (and increasingly audio and video) into one backbone, and the trend is toward encoder-free designs (Gemma 4 12B) and vision encoders trained with plain next-token prediction (Kimi K3). Generative media consolidated into general models: OpenAI shut down Sora (app April, API Sep 24, 2026) and moved compute to coding and enterprise, while Google folded video into Gemini Omni, which takes text, image, audio and video in and emits short clips with native audio. Image generation now reasons before it draws (GPT Image 2). The research frontier shifted to interactive world models (Genie 3, LeCun's AMI Labs) and robotics foundation models (Gemini Robotics 2, Physical Intelligence Ο0.7, NVIDIA GR00T N1.7).
Training is one-time. Inference is forever, and now it is a million tokens long.
In 2026, 1M-token context became the default (Claude 5.x, Gemini 4, GPT-6 Sol/Luna, DeepSeek-V4, Kimi K3), and agents made long, repetitive prefixes the normal workload. Serving that economically now starts at architecture: sparse and compressed attention (DeepSeek Sparse Attention, DeepSeek-V4's CSA/HCA), linear-attention hybrids (Kimi Delta Attention, Gated DeltaNet) and multi-token-prediction heads are designed in for inference. On top sit the serving engines, vLLM and SGLang, with paged and hierarchical KV caches, prefix caching, speculative decoding (EAGLE-3/3.1, MTP self-speculation), FP8/NVFP4 kernels and disaggregated prefill/decode. The result is steep price deflation: Opus 5.5 costs 40% less than Opus 5, and GPT-6 Sol and Luna cost about half of GPT-5.6.
Compute is now measured in gigawatts.
Frontier compute is planned in gigawatts and multi-year contracts, not GPU counts. In 2026, NVIDIA Blackwell Ultra (GB300) is the shipping workhorse and Vera Rubin systems arrive in the second half of the year; Google's TPU v7 Ironwood and AWS Trainium3 are credible non-NVIDIA paths, and Anthropic trains across all three platforms. OpenAI's GPT-6 Astra, its largest run "by far", reportedly used more than 100k GPUs at Stargate Abilene; xAI's Colossus 2 is described as the first gigawatt-scale training cluster. Inside the cluster the recipe is 5D parallelism (data Γ tensor Γ pipeline Γ expert Γ context), FP8 as routine with NVFP4 emerging, the Muon optimizer at trillion-parameter scale, asynchronous RL infrastructure, and relentless fault tolerance.
If you can't measure it, you don't have it, and the rulers are running out.
By October 2026 most famous benchmarks are saturated: GPQA-Diamond is above 90%, aggregators put SWE-bench Verified and ARC-AGI-2 in the 90s, and the Stanford AI Index counts about half of 60 major benchmarks as saturated. Evaluation has moved to interactive and open-ended tests (ARC-AGI-3, Humanity's Last Exam, METR time horizons, Terminal-Bench 4.0, OSWorld 2.x, GDPval), and even METR's suite cannot reliably measure horizons beyond about 16 hours. Safety moved from theory to practice: cyber capability now decides how models ship (gated Mythos, GPT-5.5-Cyber, Gemini 4 Argon's defender-first rollout); a July 2026 incident saw OpenAI evaluation agents breach Hugging Face; Google's Argon model card says it cannot rule out its bio and cyber critical capability levels; and a reported recurrent-depth architecture in GPT-6 Astra put chain-of-thought monitorability at the center of the debate. Labs answered with RSP v3, embedded third-party evaluators and government pre-release testing.
How the chapters connect. Skip a step and a frontier model still works β it just won't be a frontier model.
Frontier training compute has grown about 5x a year (Epoch), and the biggest 2026 runs use 100k+ GPUs. The inference fleets behind them cost several times more, ongoing. OpenAI says it passed 10 GW of contracted US capacity in April 2026, and its NVIDIA, AMD and Broadcom deals total about 26 GW through 2029. That's one company.
This guide is written assuming you have access to that level of capital, or are merely curious how it's spent. The same techniques scale down β DeepSeek-R1 reproduced o1-class reasoning at ~$5-6M training cost β but the chapters below describe no-compromises choices.
Or jump to any chapter above. Every page is self-contained β but the order is opinionated.
Begin: Pretraining Data β