Inference & Serving

Training is one-time. Inference is forever, and now it is a million tokens long.

In 2026, 1M-token context became the default (Claude 5.x, Gemini 4, GPT-6 Sol/Luna, DeepSeek-V4, Kimi K3), and agents made long, repetitive prefixes the normal workload. Serving that economically now starts at architecture: sparse and compressed attention (DeepSeek Sparse Attention, DeepSeek-V4's CSA/HCA), linear-attention hybrids (Kimi Delta Attention, Gated DeltaNet) and multi-token-prediction heads are designed in for inference. On top sit the serving engines, vLLM and SGLang, with paged and hierarchical KV caches, prefix caching, speculative decoding (EAGLE-3/3.1, MTP self-speculation), FP8/NVFP4 kernels and disaggregated prefill/decode. The result is steep price deflation: Opus 5.5 costs 40% less than Opus 5, and GPT-6 Sol and Luna cost about half of GPT-5.6.

Why it matters

KV cache, not FLOPs, is the long-context bottleneck. DeepSeek-V4 serves 1M tokens at about 27% of V3.2's per-token FLOPs and 10% of its KV cache; Kimi Linear cut KV cache by up to 75% with up to 6× decode throughput at 1M tokens.
Speculative decoding is the cheapest latency win available: vLLM's EAGLE 3.1 (May 2026) reports 2.03× per-user throughput at concurrency 1 and 1.66× at concurrency 16.
Agents re-send the same context hundreds of times, so caching pricing drives product design. Fable 5.1 cache reads cost $0.25/MTok (75% cheaper than before), and Haiku 5.5 charges $0.10/MTok input for prompts up to 100k.
Inference is now a large share of total compute spend, and labs differentiate on serving cost as much as on quality, through effort controls (Claude low/med/high/max; OpenAI none to xhigh) and paid fast modes (Opus 5.5 fast mode at $8/$40).

State of the art

2025-2026

Sparse attention for serving: DeepSeek Sparse Attention (V3.2, Dec 2025) uses a lightweight "lightning indexer" to score past tokens and attends only to the top-k, bringing attention close to O(kL); GLM-5 adopted it. DeepSeek-V4 (Apr 2026) interleaves Compressed Sparse Attention (compress KV along the sequence, then DSA) with Heavily Compressed Attention.
Linear and hybrid attention: Qwen3.5 uses a 3:1 pattern of Gated DeltaNet to gated full attention; Kimi K3 scales Kimi Delta Attention to 2.8T parameters; NVIDIA Nemotron 3 mixes Mamba-2 and Transformer layers. Constant-size recurrent state makes decode cost nearly flat in context length.
SGLang day-0 support for DeepSeek-V4 (Apr 2026): ShadowRadix prefix cache, HiSparse CPU-extended KV, and MTP speculation with in-graph metadata. LMSYS reports decode throughput falls by less than 10% from 4k to 900k tokens.
Speculative decoding: EAGLE-3 is standard and EAGLE 3.1 ships in vLLM. MTP heads trained during pretraining (DeepSeek-V3/V4, Nemotron 3, Qwen3-Next) double as built-in drafters. DeepSeek's confidence-scheduled DSpark is reported (secondary sources) at 60–85% faster per-user generation than MTP-1 on live V4 traffic.
vLLM and SGLang are the two dominant open engines. Both support paged KV, prefix/radix caching, chunked prefill, FP8 and FP4 kernels, expert parallelism for 1–3T-parameter MoEs, and prefill/decode disaggregation; TensorRT-LLM remains NVIDIA's tuned path.
Low precision: FP8 is default on Hopper and Blackwell; NVFP4 (Blackwell-native, with GEMMs about 2–3× faster than FP8 on GB200/GB300) is moving from weights-only into full inference paths.
Diffusion language models are a separate low-latency path: Inception's Mercury 2 (Feb 2026) is reported at about 1,000 tok/s on Blackwell, and NVIDIA's Nemotron-Labs-Diffusion unifies autoregressive, diffusion and self-speculative decoding in one model.
Price collapse: Opus 5.5 at $4/$20 per MTok, Sonnet 5.5 at $2/$10, Gemini 4 Argon at an introductory $2/$10 (until Nov 30, 2026), and GPT-6 Sol/Luna at about half GPT-5.6's price.

The recipe

A frontier-grade implementation, in order.

Design for serving before you train

If you control pretraining, choose attention for your target context: MLA or compressed KV, top-k sparse attention (DSA-style), or a 3:1 linear-to-full hybrid (Gated DeltaNet/KDA). Add MTP heads; they improve training signal and give you a free drafter. Retrofitting these after pretraining is much harder.

Pick a serving engine

vLLM for the broadest model and hardware support; SGLang for structured generation, radix prefix caching and day-0 support of new DeepSeek/Kimi architectures; TensorRT-LLM for maximum NVIDIA throughput. Check that the engine has kernels for your attention variant; a fallback path can be 10× slower.

Quantize

FP8 weights and activations are nearly lossless on Hopper/Blackwell. NVFP4 or MXFP4 weights on Blackwell roughly halve memory again; validate on reasoning and long-context evals, not just perplexity. Keep the KV cache in FP8 if quality holds.

Engineer the KV cache

Paged KV by default; radix or prefix caching for shared system prompts and agent histories; tier cold KV to CPU or NVMe (HiSparse-style) for million-token sessions. Expose explicit prompt-cache breakpoints to customers, since cache hit rate dominates agent economics.

Speculate

Use the model's own MTP heads first, then a trained EAGLE-3-style draft head. Tune draft length per batch size: gains are largest at low concurrency and shrink as the GPU saturates. Tree drafts help at batch 1.

Disaggregate and batch

Run prefill (compute-bound) and decode (memory-bound) on separate pools with different parallelism, transferring KV between them (DistServe, Mooncake). Continuous batching with chunked prefill keeps long prompts from stalling decode.

Expose effort as a product knob

Offer reasoning-effort tiers and a priced fast mode. Route easy requests to small models (Haiku/Flash/Luna class) and reserve frontier models for hard turns.

⚠️

Common pitfalls

"Supports 1M context" is not "works at 1M context". Measure retrieval and reasoning quality at your real lengths; sparse and linear attention can miss needles that full attention catches.

KV eviction heuristics (H2O, StreamingLLM) hurt long-context retrieval. Prefer architectural compression or tiered storage over dropping tokens at serve time.

Speculative decoding gains shrink at high batch size, and a poorly matched drafter can make throughput worse. Benchmark at your production concurrency.

Low-precision kernels vary in quality. A bad FP4 implementation can regress reasoning by several points while perplexity looks fine.

Cache economics are fragile. Bursty traffic and TTL eviction cause cold misses; co-locate requests with shared prefixes and keep system prompts byte-stable.

New architectures (DSA, CSA/HCA, KDA) need custom kernels. Until your engine supports them natively, you will not see the published efficiency numbers.