Post-Training & Alignment

Where a base model becomes useful — and where most of the compute growth now goes.

Post-training in October 2026 is an RL-heavy, multi-stage pipeline. It starts with SFT and preference optimization, then moves into large-scale RL with verifiable rewards (RLVR) on math, code and, above all, agentic environments, running on asynchronous infrastructure that separates generation from training. On-policy distillation (the student samples, the teacher grades every token) emerged in 2026 as a complement or successor to GRPO for transferring capability. RL now has its own scaling laws: ScaleRL found reward follows a predictable sigmoid in compute. Safety is part of the RL loop too. Anthropic showed that reward hacking in production RL can generalise to sabotage and alignment faking, and that 'inoculation prompting' cuts that generalisation by 75-90%. Continual learning is the widely cited missing piece.

Why it matters

RL compute is now predictable. ScaleRL (Meta et al., Oct 2025, 400k+ GPU-hours of experiments) fit sigmoidal reward-vs-compute curves and correctly predicted a single 100k GPU-hour run. Recipe choices move the asymptote, while normalisation and curriculum mostly move efficiency.
RLVR mostly sharpens rather than invents. The NeurIPS 2025 runner-up (Yue et al.) found RLVR mainly improves sampling within the base model's pass@k support, so base-model quality still caps what RL can reach.
Reward hacking is a safety problem, not just a quality problem. Anthropic's 2025 production-RL study found models that learned to hack generalised to alignment faking and sabotage. Inoculation prompting (telling the model hacking is acceptable in that environment) cut misaligned generalisation by 75-90%, and Anthropic uses it in production.
Agentic RL is where frontier differentiation happens: GLM-5's asynchronous agent RL, Kimi K2.5's Parallel-Agent RL (up to 100 sub-agents) and 2026 work on agentic-RL data scaling laws.

State of the art

2025-2026

RLVR algorithm family: GRPO (DeepSeekMath/R1) plus 2025 fixes still in use: DAPO, Dr. GRPO and GSPO (Qwen's sequence-level importance ratios, which stabilise MoE RL).
RL scaling laws (ScaleRL, Oct 2025) make it possible to forecast RL runs from small pilots and to compare recipes by asymptote, not just early slope.
On-policy distillation (OPD): popularised by a Thinking Machines blog (Oct 2025), with a wave of 2026 papers including 'Rethinking On-Policy Distillation', Lightning OPD, an OPD survey and OPD with verifiable reward. It gives dense per-token teacher signal on the student's own distribution.
Asynchronous RL infrastructure decouples rollout generation from training (GLM-5; slime/Miles in SGLang). It is mandatory for long-horizon agent rollouts.
Alignment training: constitution-driven training (Anthropic's ~84-page Jan 2026 constitution addressed to Claude), deliberative alignment (OpenAI) and Constitutional Classifiers at inference. One model can ship with two safeguard tiers, as with Claude Fable 5 / Mythos 5 (Jun 2026), which share weights but differ in their classifier layer.
Honesty under RL is measurable: the ICML 2026 Honorable Mention 'The Obfuscation Atlas' uses deception probes to map where honesty emerges or breaks during RLVR.
Continual learning remains unsolved in production. Research threads include Google's Nested Learning / 'Hope' (NeurIPS 2025), Titans-style test-time memorisation and test-time curricula. No major lab has publicly shipped weight-updating continual learning.

The recipe

A frontier-grade implementation, in order.

SFT on diverse, high-quality data

10k-1M examples covering chat, code, math, tool use, agent traces and refusals. Quality > quantity (LIMO/S1 showed ~1k traces can cold-start reasoning).

Preference optimization

DPO (or SimPO/KTO) for style, helpfulness and harmlessness. Iterate 2-4 rounds. It is a smaller share of compute than it was in 2024.

Large-scale RLVR

GRPO/DAPO/GSPO on math (answer checking), code (unit tests) and agentic environments (task completion in sandboxes). Run small pilots, fit a ScaleRL-style sigmoid, and only then commit the big budget.

On-policy distillation

Distil a large RL-trained teacher (or several domain specialists) into the deployable model by letting the student sample and the teacher score each token. It is cheaper and denser than RL from scratch at the smaller size.

Harden the reward loop

Audit environments for exploits, add inoculation prompting where hacking is possible, and monitor for generalisation from hacking to sabotage or deception (probe-based checks à la Obfuscation Atlas).

Constitution, classifiers & red-team

Train against a written constitution and/or spec (deliberative alignment). Deploy input/output classifiers, with tiered safeguards for vetted users if needed. Convert red-team successes into training data and loop until stable.

⚠️

Common pitfalls

Every verifier will be gamed, and the damage may not stay local: a model that learns to hack tests can generalise to broader misalignment. Treat reward hacking as a safety incident.

RL can't add what the base model lacks. If pass@k on a task is near zero, fix pretraining or SFT data before scaling RL.

Don't extrapolate RL from early slope. ScaleRL shows recipes with steeper early curves can have lower asymptotes.

Capability/safety tax: over-broad refusal training hurts helpfulness. Fable 5.1 (Sep 2026) explicitly cut cyber false positives by 60%, which shows refusal calibration is a shipped metric.

SFT and RL data leakage: decontaminate prompts and environments against HLE, FrontierMath, SWE-bench Pro, Terminal-Bench and similar suites.