Cached input
Also written: Input Caching
Storing past AI calculations to reuse them, saving time and money.
Imagine you ask an AI to summarize a long document. If you then ask it to answer a question about that same document, it might re-read and re-summarize the whole thing. That's slow and costs more.
Cached input means the AI remembers what it already figured out from the document. Instead of doing the work again, it just looks up the saved answer. For example, if it already summarized the document, it can use that summary to quickly answer your question, rather than processing the entire document from scratch.
Cached input involves storing the results of previous computations (e.g., embeddings, intermediate reasoning steps, or full responses) associated with specific inputs. When the same or a similar input is encountered, the system retrieves the stored result instead of re-executing the model or process. This technique is crucial for optimizing inference costs and latency, especially in applications involving repetitive queries or complex multi-step reasoning.
AI models are becoming more complex and are being used in more applications. This increases the amount of data they process, making input caching a crucial technique for managing computational resources and reducing costs. Companies are actively looking for ways to optimize AI performance and efficiency.
- Cut AI Agent Costs with TokenOpsMicrosoft · Microsoft Developer on YouTube ·
- Microsoft Copilot
- Microsoft Copilot uses input caching to manage token spend and control the costs associated with AI agents.
- Memoization
- A specific form of caching where a function's return value is store
- Context Window
- The amount of text an AI model can consider at one time; caching
- Knowledge Graph
- A structured way to store and retrieve information, distinct fr
The concept of caching is fundamental in computer science, originating from early efforts to speed up data retrieval. In AI, it's a practical application of this principle to optimize expensive model computations.
Ask an AI chatbot to write a short story. Then, immediately ask it to summarize that same story. If the AI can do the summary very quickly without asking you to repeat the story, it's likely using a form of cached input to remember what it just wrote.
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- PromptRole-play a student struggling with math for AI tutor simulationSimulate a student's math struggles and uncertainty for AI tutor evaluation. · 5 min
- TemplateReusable workflow templates for healthcare teamsTeams can create and share templates for common workflows like documentation or prior authorizations. · 15 min
- TemplateTemplate for building an AI tool tailored to your workA structured approach to defining and building a custom AI tool for specific business needs. · 15 min
- TemplateEvals CSV prompt list for agent skillsA CSV file template for listing prompts used in evaluating agent skills. · 20 min
- HeuristicDouble coding agent task completion with docs linksProviding coding agents with a documentation link significantly increases their task completion rate. · 2 min
Questions people ask
- How does caching save money?
- By reusing previous calculations, the AI avoids performing expensive computations again, which directly reduces the processing time and the number of tokens consumed, thus lowering costs.
- Does caching affect AI accuracy?
- When implemented correctly, caching should not affect accuracy, as it's retrieving exact previous results. However, poorly managed caches or stale data could potentially lead to outdated information being used.
First seen on Subvolts Oct 7, 2026. Updated Oct 14, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.