Architecture

Still a Transformer, but now efficiency-first: sparse, hybrid and very wide.

The October 2026 frontier stack is a very sparse Mixture-of-Experts Transformer with hybrid attention. Most layers use cheap attention: DeepSeek Sparse Attention, compressed KV (DeepSeek-V4's CSA/HCA), linear delta-rule layers (Gated DeltaNet, Kimi Delta Attention) or Mamba-2. A few full or latent-attention layers stay in for exact recall. The residual stream itself is now a design surface (mHC, Attention Residuals), and multi-token prediction is standard. The open disclosures set the template: DeepSeek-V4-Pro is 1.6T total / 49B active, Kimi K3 is 2.8T / 104B with 16 of 896 experts, Qwen3.5 interleaves 3 Gated DeltaNet layers per gated-attention layer, and Nemotron 3 is a Mamba-Transformer hybrid with a latent MoE. 1M-token context is the default, and these architectures exist mainly to make it affordable.

Why it matters

Long context is now an architecture problem, not a data problem. At 1M tokens, DeepSeek-V4 needs 27% of V3.2's per-token FLOPs and 10% of its KV cache, and SGLang reports V4 decode throughput drops by less than 10% from 4k to 900k tokens.
Sparsity keeps climbing. DeepSeek-V4-Pro activates ~3% of its parameters and Kimi K3 ~3.7%. Total parameter count now buys knowledge, while active count sets your serving bill.
Linear-attention hybrids finally beat full attention in fair comparisons. Kimi Linear's 3:1 KDA-to-MLA stack outperformed full attention while cutting KV cache by up to 75% and raising decode throughput up to 6x at 1M tokens.
Depth has become a knob again. Looped models (Ouro) match 2-3x larger dense models, and GPT-6 Astra (Sep 2026) is reported (not confirmed by OpenAI) to use recurrent depth. That has made architecture a CoT-monitorability question as well as an efficiency one.

State of the art

2025-2026

DeepSeek Sparse Attention (DSA, V3.2-Exp Sep 2025 → V3.2 Dec 2025): a cheap 'lightning indexer' scores past tokens and each query attends only to the top-k, bringing attention close to O(kL). GLM-5 (744B, Feb 2026) adopted it.
DeepSeek-V4 (Apr 2026, 1.6T/49B Pro, 284B/13B Flash, 1M context) interleaves Compressed Sparse Attention (compress KV along the sequence, then DSA) with Heavily Compressed Attention (more compression, dense attention). It also uses mHC residuals and Muon.
Linear/delta-rule hybrids: Gated DeltaNet sits in Qwen3-Next and Qwen3.5 (3 DeltaNet : 1 gated attention). Kimi Delta Attention adds channel-wise gating and was scaled by Kimi K3 to 2.8T parameters. Gated attention, a sigmoid gate after SDPA that removes attention sinks, won a NeurIPS 2025 Best Paper.
Hybrid SSMs: NVIDIA Nemotron 3 (Nano 31.6B Dec 2025, Super 120B-A12B Mar 2026, Ultra 550B-A55B Jun 2026, all open, 1M context on Ultra) mixes Mamba-2 and Transformer layers with LatentMoE, which routes in a low-rank latent space, plus MTP.
Residual-stream redesign: mHC (DeepSeek, Dec 2025) widens the residual into several streams with mixing matrices projected onto doubly-stochastic matrices via Sinkhorn-Knopp, restoring identity-map stability. Attention Residuals (Moonshot, Mar 2026) replace additive residuals with softmax attention over earlier layers' outputs (~2% params), used in Kimi K3.
Looped / recurrent depth: Ouro (Oct 2025) trains 1.4B and 2.6B looped LMs on 7.7T tokens that match 4B and 8-12B dense models. The gains come from better knowledge manipulation, not more storage. Reports that GPT-6 Astra uses recurrent depth remain unconfirmed by OpenAI.
Memory as a sparsity axis: Engram (DeepSeek, Jan 2026) adds hashed N-gram lookup tables. Its 'Sparsity Allocation law' finds that swapping ~20% of MoE parameters for lookup memory improves knowledge and reasoning. Whether V4 ships Engram is unconfirmed.
Multi-token prediction is standard (DeepSeek-V3/V4, Nemotron 3, Qwen3-Next). It doubles as a training signal and a built-in speculative-decoding draft head.
Diffusion LMs are a parallel track. Mercury 2 (Feb 2026) and LLaDA 2.0 (100B MoE, Dec 2025) are in production or at scale, and NVIDIA's Nemotron-Labs-Diffusion unifies AR, diffusion and self-speculative decoding. Meanwhile the ICML 2026 Outstanding Paper 'The Flexibility Trap' argues arbitrary-order decoding is worth less than assumed.

The recipe

A frontier-grade implementation, in order.

Start from a modern MoE reference

Decoder-only, RoPE, RMSNorm pre-norm, SwiGLU experts, MLA or GQA for full-attention layers, gated attention outputs. Use fine-grained routed experts plus shared experts with aux-loss-free balancing (DeepSeek-V3 lineage). Target ~3-5% active parameters at frontier scale.

Choose a hybrid attention pattern

Most layers cheap, a few exact. Proven options: 3:1 Gated DeltaNet/KDA to full or MLA attention (Qwen3.5, Kimi Linear/K3); DSA or CSA/HCA interleaving (DeepSeek-V3.2/V4); Mamba-2 + attention (Nemotron 3). Pick based on your kernel stack. Every option needs custom kernels.

Upgrade the residual stream

Ablate mHC (multi-stream residual with doubly-stochastic mixing) or Block Attention Residuals against a plain pre-norm baseline at 1-10B scale before committing. Both are now in shipped frontier open models (V4, K3).

Add MTP heads

1-2 multi-token prediction modules sharing the trunk. They give a small quality uplift and a free draft model for speculative decoding (see Inference).

Plan 1M context from day one

Pretrain at 4-8k, extend in stages to 128k → 1M with long-document mid-training. Budget KV-cache size per token as a first-class design target, not an afterthought, since that is where CSA/HCA, KDA and MLA earn their keep.

Optional bets

Engram-style lookup memory (trade MoE params for N-gram tables), looped/recurrent-depth blocks for parameter efficiency, or a diffusion decoding mode. Run each as a separate ablation track. None is required to reach the frontier.

⚠️

Common pitfalls

Don't 'innovate' on attention before you have a scaled baseline. Most clever variants regress at scale, and the ones that worked (DSA, KDA, CSA/HCA) were validated with careful scaling ablations.

Pure linear/SSM stacks still lose on exact recall (needle retrieval, long-document QA). Keep some full or latent attention layers. Every 2026 frontier hybrid does.

Extreme MoE sparsity (16 of 896 experts) is an infrastructure bet: expert parallelism, all-to-all bandwidth and load balancing dominate. MoE is still 2-3x more bug-prone than dense.

Recurrent or latent depth can reduce how much reasoning appears in the chain of thought. If you rely on CoT monitoring for safety, measure monitorability before and after (see Evaluation & Safety).

RoPE base frequency still matters for context extension. Too low and extension fails; too high and in-distribution loss bumps.