📚

Step 1 of 11

Pretraining Data

Garbage in, garbage out — at 30-trillion-token scale.

The open frontier recipes of 2026 disclose roughly 28-32T+ pretraining tokens: GLM-5 trained on 28.5T (Feb 2026) and DeepSeek-V4 on 32T+ (Apr 2026). Quality, not raw volume, is still what separates a great model from an also-ran. The pipeline starts from raw web crawl and runs through filtering, deduplication, decontamination and model-based quality scoring. The result is blended with code, math, multilingual, synthetic and, increasingly, native image/video/audio data. Agentic RL environments are now a second data supply chain sitting alongside the pretraining corpus.

Why it matters

FineWeb-Edu and DCLM showed that better filtering can be worth a 2-3x increase in compute. That lesson still holds at 30T-token scale.
Over-training keeps climbing. Llama 3 put 15T tokens into an 8B model (~1,800 tokens/param), and Ouro (Oct 2025) trained 1.4B and 2.6B looped models on 7.7T tokens.
Benchmarks are saturating fast: the Stanford AI Index 2026 counts roughly half of 60 major benchmarks as saturated. Contamination can no longer hide in a few points of noise, so decontamination is a release-blocker.
Multimodal is now part of pretraining data, not a retrofit. Qwen3.5-Omni reports over 100M hours of audio-visual data, and Kimi K3's vision encoder was trained from scratch with plain next-token prediction instead of contrastive pairs.
Agentic post-training needs its own data: verifiable environments, tool traces and multi-agent rollouts. 2026 work on agentic-RL data scaling laws treats that data as a budget line, not an afterthought.

State of the art

2025-2026

Disclosed token counts (2026 tech reports): DeepSeek-V4 32T+ (1.6T-total MoE, Apr 2026), GLM-5 28.5T (744B-total MoE, Feb 2026). Closed labs no longer publish token counts.
FineWeb / FineWeb-Edu, DCLM and Nemotron-CC remain the open baselines. LLM rephrasing of low-quality pages (Nemotron-CC, WRAP) is standard practice for recovering trillions of extra usable tokens.
Native multimodal corpora are standard. Qwen3.5 is natively vision-language, Qwen3.5-Omni adds 100M+ hours of audio-visual data, and Kimi K3 (Jul 2026) trains text, image and video together. Gemma 4's 12B model (Jun 2026) is 'encoder-free'.
Annealing on high-quality math, code and reasoning data in the last stage of pretraining (mid-training) remains universal, and long-document data is now mandatory: 1M-token context is the default for DeepSeek-V4, Kimi K3, Qwen3.8-Max and Nemotron 3 Ultra.
Synthetic data is load-bearing across the stack: rephrased web text, textbook-style rewrites, distilled reasoning traces, and RL environments generated at scale for agentic training (GLM-5's asynchronous agent RL, Kimi K2.5's Parallel-Agent RL).

The recipe

A frontier-grade implementation, in order.

Crawl & extract

Common Crawl WARC dumps → trafilatura/Resiliparse for HTML→text. Filter by language ID (fastText), domain blocklists and NSFW classifiers. Keep PDFs, code repos and long documents as separate streams, since you will need them for long-context training.

Quality classification

Label a few hundred thousand pages with a strong LLM, then distill the labels into a small classifier (the FineWeb-Edu pattern). Drop most of the web, but rephrase rather than discard the middle band (Nemotron-CC style).

Deduplicate

MinHash + LSH at document level (Jaccard ~0.7), then semantic dedup (SemDeDup) and paragraph-level near-dup removal. Expect a 60-80% reduction on raw crawl.

Decontaminate — aggressively

Do n-gram and embedding overlap checks against every eval you might report, including new suites (HLE, FrontierMath, ARC-AGI, SWE-bench Pro, Terminal-Bench). With so many benchmarks saturated, a contaminated point is a fake headline.

Mix & curriculum

Web 55-70%, code 15-25%, math 5-10%, multilingual 5-15%, books/papers 1-5%, plus interleaved image/video/audio for a native multimodal model. Fit mixture weights with small-scale ablations on your own data, then anneal on the highest-quality slice and long documents at the end.

Synthetic & environment data

Rephrase web text, generate textbook-style rewrites (Phi) and distill reasoning traces. Separately, build a library of verifiable environments (unit-tested repos, sandboxed terminals, browsers) that post-training RL will consume.

⚠️

Common pitfalls

Aggressive dedup hurts. Keep medium-similarity duplicates, since they regularize, and find the optimum with DCLM-style ablations.

Quality classifiers learn surface features like formatting and domain. Audit a held-out sample by hand.

Synthetic data collapses diversity. Rotate generators, and cap synthetic data at a fraction of the mix unless ablations prove otherwise. The NeurIPS 2025 Best Paper 'Artificial Hivemind' documents how homogeneous model outputs already are.

Benchmark contamination is now a reputational risk, not just a methodology flaw. Aggregator leaderboards already disagree by 5+ points, and contamination makes that worse.

Multilingual quality still drops off a cliff outside the top ~20 languages. Don't trust web crawl alone.

Papers to read

In rough order of foundational → recent. Click any title to open the arXiv abstract.

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

DeepSeek-AI · 2026

Apr 2026. 1.6T/49B MoE pretrained on 32T+ tokens; tech report linked from the release notes.

Kimi K3: Open Frontier Intelligence

2607.24653

Moonshot AI · 2026

Native text/image/video; vision encoder trained purely with next-token prediction.

GLM-5: from Vibe Coding to Agentic Engineering

Zhipu AI · 2026

744B/40B MoE trained on 28.5T tokens; asynchronous agent RL.

Qwen3.5-Omni Technical Report

Qwen Team (Alibaba) · 2026

Trained on over 100M hours of audio-visual data.

How Data Scales in Agentic RL: Laws and Synthesis Strategies

Various · 2026

Data scaling laws for agentic RL environments.

Scaling Latent Reasoning via Looped Language Models (Ouro)

2510.25741

ByteDance et al. · 2025

1.4B/2.6B models on 7.7T tokens — extreme over-training.

Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)

Jiang et al. · 2025

NeurIPS 2025 Best Paper.

FineWeb: decanting the web for the finest text data at scale

2406.17557

Penedo et al. (Hugging Face) · 2024

DataComp-LM: In Search of the Next Generation of Training Sets for Language Models

2406.11794

Li et al. · 2024

Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset

2412.02595

NVIDIA · 2024

WRAP: Web Rephrase Augmented Pre-training

2401.16380

Maini et al. · 2024

Scaling Synthetic Data Creation with 1,000,000,000 Personas (Persona-Hub)

2406.20094

Chan et al. · 2024

Phi-4 Technical Report

2412.08905

Microsoft · 2024

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

2401.00448

Sardana et al. · 2024

Scaling Data-Constrained Language Models

2305.16264

Muennighoff et al. · 2023

SemDeDup: Data-efficient learning at web-scale through semantic deduplication

2303.09540

Abbas et al. · 2023

The RefinedWeb Dataset for Falcon LLM

2306.01116

Penedo et al. · 2023

Training Compute-Optimal Large Language Models (Chinchilla)

2203.15556

Hoffmann et al. (DeepMind) · 2022