From text and vision to world models, robots, touch and brain interfaces: every way AI can perceive and act in the world. Updated October 2026.
Text, vision, audio and images are production-ready. Video, world models and robotics are advancing fast but are still emerging.
The foundation of modern AI. In 2026, 1M-token context became the default for frontier models, and efficient sparse and hybrid attention made long context affordable.
Understanding images, documents, screens and video frames. Vision now powers computer-use agents that operate real desktop applications.
Speech, music and environmental sound. The 2026 shift is toward omni models trained jointly on audio and video, and toward real-time interaction models that can interrupt and be interrupted.
Video generation consolidated in 2026. OpenAI shut down Sora (the app closed Apr 26 and the API on Sep 24) and moved that compute to coding and enterprise work. Google folded video into Gemini Omni at I/O (May 19); "Veo 4" has not been announced under that name.
Creating and editing images from text and reference images. The 2026 trend is models that reason before they render, which helps with layout, text in images and multi-step edits.
Models that simulate interactive, persistent environments. Google opened Genie 3 to Ultra subscribers as Project Genie (Jan 29, 2026). Yann LeCun's AMI Labs raised $1.03B (Mar 2026) to build JEPA world models, and OpenAI's former Sora team moved to world simulation for robotics.
Vision-language-action (VLA) foundation models that output motor commands. 2026 brought whole-body humanoid control and better generalization to new objects, but commercial deployment still lags research.
Standalone media apps gave way to omni models, and the frontier moved from generating pixels to simulating worlds and controlling robots.
Notable multimodal, omni, world and robot models as of October 2026, combining several modalities in a single architecture.
Generally available since Oct 13, 2026. A natively multimodal sparse MoE transformer with 1M-token context. There is a dedicated Gemini 4 Argon for Robotics variant. Its model card states Google cannot rule out several frontier-safety thresholds, and mitigations are applied.
Google's flagship after it reportedly dropped Gemini 3.5 Pro
Announced at I/O on May 19, 2026. Any-to-video: it takes text, images, audio and video as input and produces clips of up to 10 seconds with synchronized native audio and multi-shot sequences.
Video generation folded into an omni model
Released Apr 21, 2026. An image model with built-in "thinking": it reasons about the request before generating.
Reasoning before rendering
Released Sep 22, 2026 with 1M-token context. Anthropic reports 81.8% on OSWorld 2.1, where vision is used to operate real desktop applications. Text output only.
Vision as the backbone of computer-use agents
An omni model trained on over 100M hours of audio-visual data, built on the Qwen3.5 hybrid-attention family.
100M+ hours of audio-visual training data
Open weights released Jul 27, 2026: 2.8T total and 104B active parameters, with 1M context. It handles text, images and video natively. Its vision encoder was trained from scratch with next-token prediction rather than contrastive pretraining.
Largest open-weight multimodal model (as of Aug 2026)
Released Jun 3, 2026 under Apache 2.0. A "unified, encoder-free" multimodal model with no separate vision encoder, which points toward unified-token multimodality.
Open, encoder-free multimodal design
A generative world model producing explorable 720p environments at 24 fps, with about one minute of world memory. Opened to Google AI Ultra subscribers as Project Genie on Jan 29, 2026.
Playable, persistent generated worlds
Released Jul 30, 2026 with ER 2 (embodied reasoning and multi-robot planning) and On-Device 2. Demonstrated whole-body humanoid control and dexterity on Apptronik's Apollo 2.
Whole-body humanoid VLA
Released Apr 17, 2026. A steerable robot foundation model with compositional generalization: for example, it operated an air fryer it was never trained on.
Generalist robot policy for unseen tasks
An open humanoid foundation model with a dual-system design: a vision-language model plans, and a diffusion-transformer action head moves. An N2 preview is also available.
Open-weight humanoid robot model
The next frontier: sensory modalities currently in research that will expand AI's perceptual abilities. Ericsson predicts 2030 as the year of the "Internet of Senses."
Digitizing tactile sensations for robotics, VR, and prosthetics. Dexterous robot hands make touch sensing a practical bottleneck rather than a curiosity.
Digital scent technology for food, healthcare, and immersive experiences. Early research in molecular detection.
Digital taste simulation for food science, medical applications, and virtual dining experiences.
Body position and movement sensing. Critical for robotics, rehabilitation, and embodied AI, and now being tackled directly by whole-body humanoid control models.
Internal body sensing: heartbeat, breathing, hunger, temperature. Foundation for health AI.
Direct neural interfaces for thought-based control, sensory restoration, and cognitive enhancement.
Sensing radio waves, magnetic fields, and electrical signals invisible to humans.
Understanding and designing molecules, proteins and genomes. Generative biology moved from structure prediction toward design in 2026.
Human intelligence is inherently multimodal. We don't just process textβwe see, hear, touch, smell, and sense our bodies in space. The expansion of AI modalities is not just about adding features; it's about moving toward systems that understand the world as richly as we do.
Current research suggests that truly general AI will need to integrate information across all sensory modalities, understanding how sight relates to sound, how touch informs movement, and how internal states affect cognition. In 2026 the debate sharpened. One camp extends large multimodal models into world simulators and robot policies (Genie 3, Gemini Robotics 2); another, led by Yann LeCun's AMI Labs, argues that predictive world models such as JEPA are the missing piece. The journey from text-only GPT to omni-modal, embodied systems is just beginning.
FullAI provides access to cutting-edge models across text, vision, and more. Join the multimodal revolution.
Get Your Free API Key