Tokenization

The first lossy compression every model performs — and it now sets your 1M-token bill.

Tokenization controls how efficiently a model sees text, code, math and other modalities. A bad vocabulary wastes context, fragments numbers and hurts multilingual performance. In October 2026 the production default is still byte-level BPE with a 128k-256k vocabulary, but two things have changed. First, 1M-token context is now standard, so every point of compression shows up directly in KV-cache and serving cost. Second, multimodal tokenization is moving toward one unified token stream, as in Kimi K3's next-token-trained vision encoder and Gemma 4's 'encoder-free' 12B model. Learned byte-level chunking (H-Net and its 2026 follow-ups) matches BPE at research scale, but no frontier lab has publicly shipped a tokenizer-free production model.

Why it matters

Llama 3 expanded its vocab from 32k → 128k and gained ~15% throughput on the same hardware. At 1M-token context, that compression ratio multiplies straight into KV-cache memory.
Tokenization silently biases arithmetic: GPT-3.5 tokenized '1234' as one token but '12345' as two, which breaks digit-by-digit math.
Unified multimodal models need one token space across text, image, audio and video, or per-modality token costs diverge massively. Kimi K3 (Jul 2026) trained its vision encoder from scratch with next-token prediction, with no contrastive pretraining.
Tokenizer design constrains decoding speed: multi-token prediction heads and speculative drafters (now standard in DeepSeek-V4, Nemotron 3 and Qwen3-Next) predict in token space, so vocabulary size affects how cheap those heads are.

State of the art

2025-2026

Byte-level BPE (tiktoken / HF tokenizers style) with 128k-256k entries is the production default across closed and open frontier models.
Learned dynamic chunking: H-Net (Jul 2025) learns its own byte segmentation end to end. Above 1B parameters it matches BPE Transformers, and its two-stage variant scales better with data. 2026 follow-ups include SOMBRERO and adaptive targeted chunking.
BLT (Meta, Dec 2024) showed entropy-based byte patching can match BPE-trained Llama at 8B. It is still the reference point for patch-based tokenizer-free models.
No frontier lab has publicly confirmed a tokenizer-free production model as of October 2026. The real movement is in multimodality: Kimi K3 uses a next-token-trained vision encoder, and Gemma 4 12B (Jun 2026) is an 'encoder-free' unified multimodal model.
Discrete visual and audio tokenizers (FSQ for images/video, neural codecs like Mimi/EnCodec at ~12.5 Hz for speech) remain the building blocks for omni models such as Qwen3.5-Omni.

The recipe

A frontier-grade implementation, in order.

Pick an algorithm

Byte-level BPE is the safe production default. Use Unigram (SentencePiece) for heavily multilingual mixes. Prototype H-Net-style learned chunking or BLT-style patching only if you can afford a separate ablation track.

Train on representative data

Sample 1-10% of the actual pretraining mix, weighted by language and including code, math and long documents. A vocab trained on a different distribution costs throughput for the whole run.

Choose vocab size

128k-200k for English + code, 200k-300k for highly multilingual. Bigger vocab means better compression (cheaper 1M-token contexts) but larger embedding/output matrices and costlier MTP/draft heads.

Pre-tokenize digits + code

Split digits individually (or in fixed groups of three) for arithmetic robustness. Add code-aware pre-tokenization for whitespace and indentation.

Plan the multimodal token space

Reserve special tokens for image/audio/video spans, tool calls, system prompts, thinking blocks and sub-agent messages. Decide early whether vision enters as a separate encoder or as unified next-token-trained tokens (the Kimi K3 / Gemma 4 12B direction).

⚠️

Common pitfalls

Don't reuse another model's tokenizer for a new pretraining run. Vocab mismatch costs throughput and quality.

Glitch tokens (SolidGoldMagikarp etc.) emerge from rare merges. Audit the long tail of your vocab.

Tokenizer changes are not backwards-compatible: switching invalidates cached prompts, KV caches, MTP heads and speculative drafters.

Byte-level fallback is critical. Never crash on unseen bytes such as emoji or rare scripts.

Don't bet a frontier run on tokenizer-free models yet. H-Net-style results are at the 1B-scale research stage, and no lab has disclosed one in production.