Multi-Token Prediction
Also written: MTP
AI generates text faster by predicting and outputting multiple words or tokens at once, instead of one by one.
Imagine an AI writing a story. Normally, it writes one word, then thinks about the next word, then writes that word, and so on. This is like a person typing one letter at a time.
Multi-token prediction is like that person typing whole words or even short phrases at a time. The AI predicts and outputs several words together. This makes the writing process much faster, especially for longer pieces of text.
Multi-token prediction (MTP) is an inference optimization technique where a language model generates multiple output tokens in a single forward pass, rather than sequentially. This contrasts with standard autoregressive generation, which predicts one token at a time. MTP methods can significantly increase generation throughput, but may involve trade-offs in model complexity, latency for the first token, or generation quality if not carefully implemented.
Recent advancements in model architectures and optimization techniques are making multi-token prediction more practical. Companies are exploring these methods to improve the speed and efficiency of their large language models, especially as model sizes continue to grow.
- See what 3 builders are making with Gemma 4Google · The Keyword: Developers ·
- Google Gemini
- Google's Gemma models are being used by developers, and MTP is a common technique to speed up their generation capabilities.
- Speculative Decoding
- This technique uses a smaller, faster model to predict sev
- Batching
- This involves processing multiple independent requests simultaneously
- Beam Search
- This is a decoding strategy that explores multiple possible token s
The concept of generating multiple tokens simultaneously has evolved over time with research into efficient inference for large language models. Specific techniques and optimizations are often detailed in research papers from various institutions and companies.
You can observe faster text generation in many AI chatbots. Try asking a model like Google Gemini or ChatGPT to write a short story or a long email. Notice how quickly the text appears on your screen compared to older AI text generators.
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- PromptRole-play a student struggling with math for AI tutor simulationSimulate a student's math struggles and uncertainty for AI tutor evaluation. · 5 min
- TemplateReusable workflow templates for healthcare teamsTeams can create and share templates for common workflows like documentation or prior authorizations. · 15 min
- TemplateTemplate for building an AI tool tailored to your workA structured approach to defining and building a custom AI tool for specific business needs. · 15 min
- TemplateEvals CSV prompt list for agent skillsA CSV file template for listing prompts used in evaluating agent skills. · 20 min
- HeuristicDouble coding agent task completion with docs linksProviding coding agents with a documentation link significantly increases their task completion rate. · 2 min
Questions people ask
- Does multi-token prediction always make AI faster?
- It significantly increases generation throughput, meaning more tokens can be produced per unit of time. However, the latency for the very first token might sometimes be slightly higher.
- Can multi-token prediction affect the quality of the text?
- Yes, if not implemented carefully, it can sometimes lead to lower quality or less coherent text. Researchers are developing methods to balance speed and quality.
First seen on Subvolts Jun 9, 2026. Updated Sep 15, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.