Multimodal

Native any-to-any is the default; the frontier moved to worlds and robots.

Every 2026 frontier model is natively multimodal: Gemini 4 Argon, Meta's Muse Spark, Kimi K3 and Qwen3.5 all train text and vision (and increasingly audio and video) into one backbone, and the trend is toward encoder-free designs (Gemma 4 12B) and vision encoders trained with plain next-token prediction (Kimi K3). Generative media consolidated into general models: OpenAI shut down Sora (app April, API Sep 24, 2026) and moved compute to coding and enterprise, while Google folded video into Gemini Omni, which takes text, image, audio and video in and emits short clips with native audio. Image generation now reasons before it draws (GPT Image 2). The research frontier shifted to interactive world models (Genie 3, LeCun's AMI Labs) and robotics foundation models (Gemini Robotics 2, Physical Intelligence π0.7, NVIDIA GR00T N1.7).

Why it matters

Bolt-on adapters are legacy. Native multimodal pretraining is now standard at every frontier lab, and the open weights to study it exist (Gemma 4, Qwen3.5-Omni, Kimi K3).
Dedicated media products lost to general models: Sora's shutdown and Gemini Omni's any-to-video design show video generation becoming a capability of the main model rather than a separate product line.
World models are the bet for embodied AI. Genie 3 generates navigable 720p worlds at 24 fps, AMI Labs raised a $1.03B seed to build JEPA world models, and OpenAI moved its Sora team to world simulation for robotics.
Robotics foundation models now generalize compositionally: π0.7 operated an air fryer it was never trained on, and Gemini Robotics 2 runs whole-body control on a humanoid. Commercial deployment still lags the research demos.

State of the art

2025-2026

Gemini 4 Argon (GA Oct 13, 2026): sparse MoE, natively multimodal, 1M context. A Gemini 4 Argon for Robotics variant is also offered to developers.
Gemini Omni Flash (Google I/O, May 19, 2026): any-to-video model taking text, image, audio and video, producing clips up to 10 s with synchronized native audio and multi-shot sequences. Google has not shipped a model under the name "Veo 4".
GPT Image 2 (OpenAI, Apr 21, 2026): image generation with built-in reasoning before rendering. Nano Banana 2 (Gemini 3.1 Flash Image, Feb 26, 2026) is Google's counterpart.
Sora shutdown: OpenAI closed the Sora app on Apr 26 and the API on Sep 24, 2026, reallocating compute to coding and enterprise and moving the team to world simulation for robotics.
Unified open models: Qwen3.5-Omni (trained on over 100M hours of audio-visual data), Gemma 4 12B ("unified, encoder-free", Jun 2026), and Kimi K3 (2.8T open MoE whose vision encoder is trained from scratch with next-token prediction, no contrastive pretraining).
Real-time interaction: Thinking Machines' interaction models can interrupt mid-dialogue; full-duplex speech (Moshi lineage) is the template.
World models: Genie 3 opened to Google AI Ultra subscribers as Project Genie (Jan 29, 2026) with 720p/24 fps output and about a minute of visual memory. AMI Labs (Yann LeCun, $1.03B seed, Mar 2026) pursues JEPA-style predictive world models instead of pixel generation. Benchmarks such as WorldRoamBench are starting to appear.
Robotics: Gemini Robotics 2 (Jul 30, 2026) adds whole-body humanoid control on Apptronik Apollo 2, ER 2 for embodied reasoning and multi-robot coordination, and On-Device 2. π0.7 (Physical Intelligence, Apr 17, 2026) is steerable with compositional generalization. NVIDIA Isaac GR00T N1.7 is open, with an N2 preview, using a dual-system design: a VLM planner plus a diffusion-transformer action head.

The recipe

A frontier-grade implementation, in order.

Choose the visual front end

Late fusion with a SigLIP 2 encoder is still the cheapest retrofit. For new pretrains, the 2026 trend is native fusion: either an encoder-free design (Gemma 4 12B) or an encoder trained jointly with next-token prediction (Kimi K3). Use NaViT-style native resolution and aspect ratio.

Pretrain multimodal from the start

Interleave image, video and audio tokens with text throughout pretraining rather than bolting on vision at the end. Reweight the loss so text does not dominate, and spend heavily on synthetic recaptioning of image and video data.

Audio and real-time voice

Neural codec tokens (Mimi/EnCodec-style RVQ at 12.5–25 Hz) interleaved with text for full-duplex dialogue. Train explicit turn-taking and interruption behavior; latency is a product requirement, not a nice-to-have.

Generation heads

Diffusion transformers over a spacetime VAE latent for video, with joint audio synthesis. Let the language backbone plan first: GPT Image 2 and Gemini Omni condition generation on the model's own reasoning and on any-modality input.

World models

Two schools: action-conditioned generative video you can walk around in (Genie 3), or latent predictive models that never render pixels (JEPA, V-JEPA 2, AMI Labs). Evaluate on consistency over time and controllability, not visual quality alone.

Vision-language-action for robots

Start from a pretrained VLM, add an action expert (flow matching or a diffusion head as in π0 and GR00T), and train on cross-embodiment teleop data plus web data. Separate slow reasoning (an ER-style planner) from fast control; distill to on-device models for latency.

Evaluate

MMMU and ChartQA for vision QA, VBench plus human preference for video, image-arena preference for images, and real-robot success rates on held-out tasks and embodiments. Simulation-only robot scores overstate real-world performance.

⚠️

Common pitfalls

Data is the constraint. Clean image-text and video-text pairs are scarcer than text, and audio-visual corpora at the 100M-hour scale are hard to license. Synthetic recaptioning helps but can collapse diversity.

Modality imbalance: text dominates the loss unless you reweight, and vision skills regress silently during text-heavy post-training. Track multimodal evals through every post-training stage.

Standalone media generation is expensive and strategically fragile; Sora's shutdown is the cautionary tale. Budget inference cost per second of video before shipping.

World-model demos are short. Genie 3 holds roughly a minute of consistent memory. Long-horizon consistency and physical accuracy remain unsolved.

Robot demos are not deployments. Gaps between lab success rates and messy real-world environments remain large, and cross-embodiment transfer is uneven.