Speculative decoding
Speculative decoding speeds up AI text generation by having a small model predict ahead, then a large model verifies.
Imagine you're asking an AI to write a story. Normally, it writes one word at a time, and you wait for each word to appear before it writes the next.
Speculative decoding is like having a fast assistant who quickly guesses the next few words. Then, the main, more powerful AI checks if those guesses are good. If they are, it accepts them all at once, saving time. If not, it corrects the guesses and tries again. This makes the AI feel much faster when it's generating text.
Speculative decoding is an inference technique designed to accelerate the generation speed of large language models (LLMs). It employs a smaller, faster "draft" model to generate a sequence of candidate tokens in parallel. A larger, more accurate "verification" model then processes these candidates in a single forward pass to accept or reject them. This approach can significantly reduce the number of forward passes required, thereby lowering latency and increasing throughput, especially when the draft model's predictions are often correct. Tradeoffs include the overhead of running two models and the complexity of managing the draft-verification process.
Speculative decoding is appearing in news as companies like Google Gemini focus on making AI models generate responses faster. This technique is key to improving the user experience of AI assistants by reducing latency, especially for complex tasks that require generating longer outputs.
- GPT-6.1 Ultrafast ⚡, Gemini universal agent 🤖, speculative decoding 🧠Google · TLDR AI ·
- Yossi Matias on the golden age of researchGoogle · Google AI: Release Notes ·
- Google Gemini
- Google Gemini is using speculative decoding to accelerate text generation in its large language models, aiming to make AI assistants more powerful and responsive.
- Model Distillation
- This process creates a smaller model that mimics a larger on
- Caching
- Caching stores previously computed results to avoid redundant calculati
- Parallel Decoding
- This refers to generating multiple possible sequences simulta
The concept of speculative decoding was formally introduced in a 2023 paper by researchers from Google, aiming to accelerate LLM inference. Source ↗
While direct experimentation with speculative decoding is complex, you can observe its effect by comparing the response times of different AI models. Try asking the same prompt to a very large, powerful model and a smaller, faster model. You might notice that models optimized with techniques like speculative decoding respond more quickly, especially for longer outputs.
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- PromptRole-play a student struggling with math for AI tutor simulationSimulate a student's math struggles and uncertainty for AI tutor evaluation. · 5 min
- TemplateReusable workflow templates for healthcare teamsTeams can create and share templates for common workflows like documentation or prior authorizations. · 15 min
- TemplateTemplate for building an AI tool tailored to your workA structured approach to defining and building a custom AI tool for specific business needs. · 15 min
- TemplateEvals CSV prompt list for agent skillsA CSV file template for listing prompts used in evaluating agent skills. · 20 min
- HeuristicDouble coding agent task completion with docs linksProviding coding agents with a documentation link significantly increases their task completion rate. · 2 min
Questions people ask
- How does speculative decoding make AI faster?
- It uses a small, quick model to guess ahead. A larger, more accurate model then checks these guesses efficiently, reducing the total work needed to generate text.
- Does speculative decoding affect the AI's output quality?
- Ideally, no. The larger, verifying model ensures that the final output is still accurate and coherent, maintaining the quality of the primary model.
First seen on Subvolts Jun 22, 2026. Updated Oct 15, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.