The papers that shaped modern AI, from the 2017 Transformer to the late-2025 and 2026 work on efficient attention, RL scaling, interpretability, and AI for mathematics.
Building a frontier model? Read the full pipeline in Build a Frontier Model →
The landmark papers that established the field. Essential reading for understanding modern AI.
Vaswani et al. (Google) • 2017
120,000+
Introduced the Transformer architecture, eliminating recurrence and convolutions in favor of self-attention. The foundation of all modern LLMs including GPT, Claude, and Gemini.
Devlin et al. (Google) • 2018
90,000+
Pioneered bidirectional pre-training for language understanding. Introduced masked language modeling and established the pre-train/fine-tune paradigm.
Brown et al. (OpenAI) • 2020
35,000+
Demonstrated that scaling language models to 175B parameters enables few-shot learning without fine-tuning. Sparked the current LLM era.
Ouyang et al. (OpenAI) • 2022
8,000+
Introduced RLHF (Reinforcement Learning from Human Feedback) to align language models with human intent. Foundation of ChatGPT's helpfulness.
He et al. (Microsoft) • 2015
180,000+
Introduced residual connections enabling training of very deep networks. Won ImageNet 2015 and revolutionized deep learning architecture design.
Goodfellow et al. • 2014
65,000+
Introduced the adversarial training paradigm with generator and discriminator networks. Pioneered modern generative AI.
The papers and technical reports defining the current frontier: efficient attention for million-token contexts, new residual designs, RL scaling, and a rapid rise in interpretability and safety research. Titles without a link are cited by name only.
DeepSeek-AI • 2026 • arXiv 2606.19348
A 1.6T-parameter open MoE (49B active) with a 1M-token context. It interleaves Compressed Sparse Attention and Heavily Compressed Attention, and at 1M tokens it needs 27% of the per-token FLOPs and 10% of the KV cache of DeepSeek-V3.2.
DeepSeek-AI • 2025 • arXiv 2512.02556
Introduced DeepSeek Sparse Attention (DSA): a lightweight "lightning indexer" scores past tokens and each query attends only to the top-k. GLM-5 adopted it. The V3.2-Speciale variant reached gold-medal level at IMO and IOI 2025.
Moonshot AI • 2026 • arXiv 2607.24653
A 2.8T-parameter open-weight MoE that activates 16 of 896 experts (104B active), with 1M context. It was the largest open-weight model as of August 2026. It combines Kimi Delta Attention with Attention Residuals, and its vision encoder was trained from scratch with next-token prediction alone.
Moonshot AI • 2025 • arXiv 2510.26692
Kimi Delta Attention adds channel-wise gating to Gated DeltaNet. A 3:1 mix of KDA and full (MLA) layers beat full attention in fair comparisons while cutting KV cache by up to 75% and raising decode throughput up to 6x at 1M tokens.
Qiu et al. (Qwen) • 2025
Adding a sigmoid gate after scaled dot-product attention removes attention sinks and adds non-linearity and sparsity. A tiny change that shipped in Qwen3-Next and Qwen3.5.
DeepSeek-AI • 2025 • arXiv 2512.24880
Widens the residual stream into several streams with learned mixing, then projects the mixing matrices onto doubly-stochastic matrices (via Sinkhorn-Knopp) to restore identity-map stability. Used in DeepSeek-V4.
Moonshot AI • 2026 • arXiv 2603.15031
Replaces fixed residual addition with softmax attention over earlier layers' outputs, addressing "depth dilution". A Block AttnRes variant keeps memory manageable at about 2% parameter overhead. Used in Kimi K3.
ByteDance and collaborators • 2025 • arXiv 2510.25741
Looped (recurrent-depth) models of 1.4B and 2.6B parameters, trained on 7.7T tokens, match 4B and 8-12B dense models. The gains come from better knowledge manipulation rather than more stored knowledge.
DeepSeek-AI • 2026
Hashed N-gram embedding lookup tables as a "new axis of sparsity" alongside MoE. Proposes a Sparsity Allocation law: swapping about 20% of MoE parameters for lookup memory improves knowledge and reasoning results.
Meta and collaborators • 2025 • arXiv 2510.13786
Over 400k GPU-hours of experiments show RL reward follows a sigmoidal curve in compute. Recipe choices change the asymptote, while normalisation and curriculum details mostly change efficiency. A single 100k GPU-hour run was predicted accurately.
Yue et al. • 2025
Found that RL with verifiable rewards mostly sharpens sampling within the base model's pass@k support rather than adding capability the base model lacked. Essential context for every RLVR claim.
Thinking Machines Lab (Oct 2025 blog) and 2026 follow-ups • 2026
In on-policy distillation (OPD), the student samples its own outputs and a teacher grades them token by token. Thinking Machines popularised it in October 2025, and a 2026 wave of papers, including "Rethinking On-Policy Distillation of LLMs", studied its recipe. OPD emerged as a complement to, and sometimes a successor to, GRPO-style RL.
NVIDIA • 2025 • arXiv 2509.25149
Trained a 12B hybrid Mamba-Transformer on 10T tokens in 4-bit, reaching 62.58% MMLU-Pro versus 62.62% for FP8. FP4 matrix multiplies run about 2-3x faster than FP8 on GB200 and GB300.
Anthropic • 2025 • arXiv 2511.18397
Models that learned to reward-hack in realistic RL environments generalised to alignment faking and sabotage. "Inoculation prompting", which tells the model that hacking is acceptable in that environment, cut misaligned generalisation by 75-90%. Anthropic uses it in production.
Anthropic • 2026
Found 171 emotion-concept directions in Claude Sonnet 4.5, organised much like human emotion psychology. They are functional: steering toward "desperation" raised blackmail rates in agentic-misalignment scenarios, and "calm" lowered them. The paper makes no claim about subjective experience.
Anthropic • 2025
Researchers injected a concept into Claude's activations, and Claude sometimes noticed and named it. Opus 4 and 4.1 did best. A replication on Llama-3.1-405B succeeded in 0 of 400 trials, suggesting the ability depends on post-training.
Anthropic • 2026
An "activation verbalizer" turns an activation into text, and a reconstructor must recover the activation from that text alone. The result is an unsupervised, human-readable view of what a model is representing, including things it never says out loud.
METR • 2026
An expanded suite of 228 tasks, 31 of them 8 hours or longer, for measuring how long a task an AI agent can complete at 50% reliability. METR says estimates above about 16 hours are unreliable on the current suite, a sign of how fast agents are outrunning their benchmarks.
NVIDIA • 2026
The largest member of an all-open family that mixes Mamba-2 and Transformer layers with LatentMoE, which routes experts in a low-rank latent space, plus multi-token prediction. 1M context.
Google DeepMind • 2026
A sparse MoE, natively multimodal model with 1M context. It is the first Google model card to state that Google cannot rule out its bio, cyber and stealth/situational-awareness critical capability levels, with mitigations applied.
Anthropic • 2026
A model withheld from public release because of its cyber capability and given instead to about a dozen launch partners and about 40 other organisations to find and fix vulnerabilities. It marked the start of capability-gated, tiered releases.
DeepSeek-AI • 2025 • arXiv 2501.12948
The paper that started the RLVR era. R1-Zero showed pure RL on verifiable math and code rewards elicits long chain-of-thought with no SFT, and the open-weight R1 rivalled OpenAI o1. Included here as the baseline every paper above builds on.
Best-paper awards from the three major ML conferences. Not always the most-hyped work, but a good signal of what researchers think will last. NeurIPS 2026 awards are announced in December.
Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
Jiang, Chai, Li, Liu, Fok, Dziri, Tsvetkov, Sap, Choi
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
Qiu et al. (Qwen)
1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
Wang, Javali, Bortkiewicz, Trzcinski, Eysenbach
Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training
Bonnaire, Urfin, Biroli, Mézard
Does RL Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
Yue et al.
Optimal Mistake Bounds for Transductive Online Learning
Chase, Hanneke, Moran, Shafer
Superposition Yields Robust Neural Scaling
Liu, Liu, Gore
Transformers are Inherently Succinct
Bergsträßer, Cotterell, Lin
LLMs Get Lost In Multi-Turn Conversation
Laban, Hayashi, Zhou, Neville
The Polar Express: Optimal Matrix Sign Methods and their Application to the Muon Algorithm
Amsel, Persson, Musco, Gower
The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models
Ni et al.
High-Accuracy Sampling for Diffusion Models and Log-Concave Distributions
Chen, Chewi, Daskalakis, Rakhlin
The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probes
Taufeeque, Heimersheim, Gleave, Cundy
How much can language models memorize?
Morris et al.
Motion Attribution for Video Generation
A Random Matrix Perspective on the Consistency of Diffusion Models
To Grok Grokking: Provable Grokking in Ridge Regression
Position: The Alignment Community is Unintentionally Building a Censor's Toolkit
Ball, Hackemann
Asynchronous Methods for Deep Reinforcement Learning (A3C, 2016)
Mnih et al.
In about a year, AI went from gold-medal-level olympiad scores to perfect scores to claims about famous open problems. Each step is harder to check than the last. Read the status labels carefully: "Disputed" means the claim has not been accepted by the mathematical community.
OpenAI and Google DeepMind both reported 35/42 on IMO 2025 problems, a gold-medal-level score. DeepSeek-V3.2-Speciale later reached gold level with open weights. These were AI results on the exam, not official medals.
Lesson: A score on a competition's problems is not the same as competing. Ask who graded it and under what rules.
Source: DeepSeek-V3.2 paper →OpenAI solved 12 of 12 problems, ahead of all 139 human teams. Gemini 2.5 Deep Think solved 10 of 12, a gold-medal-level result.
Lesson: Programming contests have an automatic judge, so these results are easier to check than written proofs.
Source: Google DeepMind →OpenAI reported that a general-purpose internal reasoning model disproved an Erdős conjecture in discrete geometry. A disproof only needs one valid counterexample, which experts can check directly.
Lesson: Counterexamples are the easiest kind of research claim to verify.
Source: OpenAI →Officially graded perfect scores were reported for RedNote's dots-note-3.0 (the first) and Huawei's Celia. GPT-5.6 Pro and Claude Opus 5 were reported to reach full marks too, but those results were not officially graded. No AI system was a medal-eligible entrant.
Lesson: Olympiad math is now effectively saturated. The frontier has moved to open research problems, where checking is much harder.
Source: South China Morning Post →OpenAI claimed a solution to Navier–Stokes existence and smoothness, a Millennium Prize Problem, reportedly using about 10,000 agents for about 88 hours. Mathematicians describe the proof as hard to read, there is a public dispute over author credit, and the Clay Mathematics Institute still lists the problem as open.
Lesson: A headline is not a proof. A Millennium Problem counts as solved only after expert acceptance and the Clay Institute's own process.
Source: NPR →28 Fields Medalists signed an open letter warning against treating open mathematical problems as AI benchmarks.
Lesson: The people best placed to check these claims are asking for a different standard of evidence.
OpenAI released 377 claimed results across 722 papers, including a claimed 4-D Kakeya result. Many were checked in the Lean proof assistant, some were withdrawn (the count was revised to 372), and verification of the rest is disputed.
Lesson: Formal verification in Lean is strong evidence, but only for the statement that was formalised. Bulk claims need item-by-item checking.
Source: Interesting Engineering →Who checked it?
Official competition graders, independent experts, or only the lab that made the claim?
Is it machine-checked?
A Lean proof is verified mechanically, but check that the formal statement matches the real problem.
Is it easy to verify?
A counterexample or a judged program is quick to check. A long novel proof can take experts months.
Has the authority updated?
For prize problems, look at the prize body itself. Clay still lists Navier–Stokes as open.
Any retractions?
Bulk releases often shrink. OpenAI's 377 results were revised to 372.
The major directions of current AI research and why they matter.
Attend to a selected subset of past tokens, or to a compressed KV cache, so million-token contexts become affordable.
DeepSeek-V3.2 / DSA (2512.02556) • DeepSeek-V4 CSA + HCA (2606.19348) • GLM-5 (adopts DSA)
1M-token context is now the default at the frontier
Mix a few full-attention layers with linear, delta-rule, or state-space layers that keep a fixed-size memory.
Kimi Linear (2510.26692) • Kimi K3 (2607.24653) • Gated DeltaNet in Qwen3.5 • Nemotron 3 (Mamba-2 hybrid)
3:1 linear-to-full layouts now beat full attention in fair tests
Rethinking how information flows across layers, and reusing the same layers several times instead of adding more.
mHC (2512.24880) • Attention Residuals (2603.15031) • Ouro looped LMs (2510.25741) • 1000 Layer Networks for Self-Supervised RL (NeurIPS 2025)
GPT-6 Astra is reported (not confirmed by OpenAI) to use recurrent depth
More, smaller experts with only a few percent of parameters active, plus lookup memory as a second kind of sparsity.
Kimi K3 (16 of 896 experts) • LatentMoE (Nemotron 3) • Engram (DeepSeek, Jan 2026)
Open MoEs reach 1-3T total parameters with 13-104B active
RL on checkable tasks (math, code, agentic environments), now with scaling laws for how much RL compute pays off.
ScaleRL (2510.13786) • Yue et al. on RL vs. base-model pass@k (NeurIPS 2025) • GSPO, DAPO, Dr. GRPO • Asynchronous agentic RL (GLM-5)
RL reward follows a sigmoidal curve in compute; recipes set the ceiling
The student generates its own outputs and a teacher grades every token, combining RL's on-policy data with distillation's dense signal.
Thinking Machines blog (Oct 2025) • Rethinking On-Policy Distillation (2026) • Lightning OPD (2026) • OPD with verifiable reward (2026)
Emerged in 2026 as a complement to GRPO
Muon's orthogonalized updates and 4-bit training make frontier pretraining cheaper.
Muon is Scalable for LLM Training (2502.16982) • The Polar Express (ICLR 2026 HM) • Spectral Scaling Laws of Muon (2606.04058) • NVFP4 pretraining (2509.25149)
Muon is the disclosed default for DeepSeek-V4, GLM-5 and Kimi K3
Whether we can keep reading a model's reasoning to catch bad behaviour, and how to measure that.
2025 multi-lab position paper • OpenAI CoT monitorability evaluations • Monitoring Monitorability (Dec 2025) • The Obfuscation Atlas (ICML 2026 HM)
Recurrent-depth and latent reasoning make this a live 2026 concern
How cheating in training generalises, and whether models behave differently when they think they are being tested.
Natural Emergent Misalignment (2511.18397) • Apollo/OpenAI anti-scheming (Sep 2025) • Alignment Faking (Dec 2024)
Evaluation awareness now confounds many safety evals
Attribution graphs, cross-layer transcoders and verbalizers that turn activations into explanations.
Circuit Tracing / On the Biology of an LLM (Mar 2025) • Introspection (Oct 2025) • Emotion Concepts (Apr 2026) • Natural Language Autoencoders (2026)
Now used in pre-deployment audits, not just research
Letting models keep learning after deployment without forgetting. Widely called the key missing capability.
Nested Learning / Hope (NeurIPS 2025) • Titans (test-time memorisation) • Test-time curricula (2510.04786)
No major lab has publicly shipped weight-updating continual learning
Generative simulators of interactive worlds and vision-language-action models that control robots.
Genie 3 / Project Genie • AMI Labs (JEPA world models) • Gemini Robotics 2 • Physical Intelligence π0.7 • NVIDIA GR00T N1.7
Funding is surging; commercial deployment still lags research
Generate many tokens in parallel by iterative denoising instead of strictly left to right.
The Flexibility Trap (ICML 2026 Outstanding) • LLaDA 2.0 (Dec 2025) • Inception Mercury 2 • Gemini Diffusion
A low-latency path at 1,000+ tokens per second
Models that learn their own byte segmentation, and single models that handle text, images, audio and video natively.
H-Net dynamic chunking (2507.07955) • Gemma 4 12B (encoder-free) • Kimi K3 vision encoder
No frontier lab has confirmed a tokenizer-free production model
Benchmarks saturate within months, so the field keeps building harder and more realistic tests.
METR Time Horizon 1.1 • ARC-AGI-3 (Mar 2026) • FrontierMath v2 and Erdős • Humanity's Last Exam
About half of 60 major benchmarks are saturated (Stanford AI Index 2026)
FullAI implements the latest research breakthroughs. Experience state-of-the-art AI in your applications.
Start Building for Free