Top news
Investing in the public benefit OfficialAnthropic NewsOct 18Popular Musk Pledges SpaceX Shares to ‘Every Kid in America’ WebBloomberg TechnologyOct 19Popular Websites are closing their doors to AI agents WebThe Verge AIOct 17 This is the new ChatGPT Voice, powered by GPT Live OfficialOpenAI on YouTubeOct 18 How AI giants will reshape global philanthropy WebAxios AIOct 19 Gemini 4 Argon Really Surprised Me WebTwo Minute PapersOct 19 Google Gemini 4 agents just beat Minecraft with no human intervention WebEngadget AIOct 19 Who could it be? 🧐 OfficialGoogle on YouTubeOct 18 Anthropic gives outside AI watchdog “employee-like” access WebArs Technica AIOct 18 What are self-replicating ‘AI worms’, and why is Anthropic worried about them? WebThe Guardian AIOct 18
GlossaryModelsIntermediate

Multimodal model

Also written: Multimodal AI, Multimodal LLMs, MLLMs, Multimodal Large Language Models, VLMs, Multi-modal temporal-spatial vision transformer, MTSViT, Multi-modal models

AI that understands and processes multiple types of data like text, images, audio, and video.

In plain terms

A multimodal AI model can understand and work with different kinds of information at the same time. Think of it like a person who can read a book, look at the pictures, and listen to an audiobook all at once.

Instead of just understanding text, these AI models can also process images, sounds, and even video. For example, you could show a multimodal AI a picture of a cat and ask it to describe the cat in text, or show it a video and ask it to summarize what's happening.

The expert version

Multimodal models are AI systems designed to process and integrate information from diverse data modalities, including text, images, audio, and video. This allows them to perform tasks that require understanding relationships across different data types, such as generating captions for images, answering questions about visual content, or transcribing audio while considering visual cues.

The integration of multiple modalities typically involves specialized encoders for each data type, followed by fusion mechanisms that combine the representations. This enables richer contextual understanding and more sophisticated reasoning capabilities compared to unimodal models. Tradeoffs often involve increased computational complexity and the challenge of aligning representations from disparate data sources.

Why it matters now

Multimodal models are a key focus as frontier AI labs aim to create more versatile and human-like AI systems. Recent advancements allow these models to handle complex tasks involving multiple data types, as seen in Meta's SAM Audio for sound separation and Google's Gemini for classifying astronomical events.

Where it shows up 15 items, newest first
How each company uses it
Google Gemini
Google's Gemini models are designed to be multimodal, capable of processing and understanding text, images, audio, and video.
Meta Meta AI
Meta AI has developed multimodal models like SAM Audio, which can process audio using text or visual prompts.
OpenAI ChatGPT
OpenAI's GPT-4 is described as a multimodal AI model, indicating its ability to handle various data types.
Related terms

Large language modelArtificial intelligenceComputer visionNatural language processingGenerative AIFoundation model

Often confused with
Large language model
LLMs primarily process and generate text, while multimodal
Computer vision model
Computer vision models focus specifically on analyzing im
Speech recognition model
These models are specialized for converting audio to t
Origin

The concept of multimodal AI has roots in early AI research aiming to mimic human perception, which naturally integrates multiple senses. The term gained significant traction with the development of deep learning architectures capable of processing diverse data streams.

Try this about 5 minutes

Try using a multimodal AI tool like Google Gemini or ChatGPT. Upload an image and ask the AI to describe it in detail, or ask it to write a story based on a picture you provide. This demonstrates the AI's ability to connect visual input with text generation.

Try it: Playbook entries that use thisPlaybook →

The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.

Questions people ask

Can multimodal AI see and hear?
Yes, advanced multimodal AI models can process visual information from images and videos, and auditory information from sound and speech.
What's the difference between AI and multimodal AI?
AI is a broad field. Multimodal AI is a specific type of AI that can understand and use multiple kinds of data, not just one.

First seen on Subvolts Mar 14, 2023. Updated Oct 12, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.