Throughput
Also written: Provisioned Throughput Units, PTUs
How much work an AI system can do in a set amount of time.
Throughput measures how much work an AI system can complete in a specific period. Think of it like a factory's production line: higher throughput means more items are made in the same amount of time.
For AI, this could mean how many text requests it can answer, how many images it can generate, or how much data it can process. Faster throughput means the AI can handle more tasks or serve more users simultaneously.
Throughput quantifies the rate at which a computational system, such as an AI model or inference engine, can process tasks or data. It is typically measured in units per second, such as queries per second (QPS) or tokens per second. High throughput is crucial for applications requiring real-time responses or processing large volumes of data efficiently.
Optimizing throughput often involves balancing computational load, memory access, and network latency. Tradeoffs exist between throughput, latency (the time for a single task), and cost. Systems designed for high throughput may sacrifice individual task speed for overall processing capacity.
Frontier AI labs are increasingly focused on optimizing the cost and performance of AI models, especially for high-volume agent tasks. This involves managing compute capacity efficiently, with terms like throughput and provisioned throughput units (PTUs) becoming important for balancing performance and economics. Companies are developing specialized models and strategies to ensure they can handle large workloads cost-effectively.
- Higgsfield - Building Agents That Provision Their Own Compute | DevDay 2026OpenAI · OpenAI on YouTube ·
- bolt Gemini 3.5 Flash-LiteGoogle · Gemini Models ·
- bolt Gemini 3.5 Flash-LiteGoogle · Gemini Models ·
- The Economics of Agent Optimization: Four ways to lower the costMicrosoft · Microsoft Azure Blog ·
- The full stack behind abundant intelligenceOpenAI · OpenAI News ·
- LWiAI Podcast #236 - GPT 5.4, Gemini 3.1 Flash Lite, Supply Chain RiskGoogle · Last Week in AI ·
- Google Gemini
- Google is releasing faster, lower-cost Gemini models like Flash-Lite for high-volume agent tasks and document analysis, aiming for better throughput and economics.
- Microsoft Copilot
- Microsoft discusses strategies to optimize AI agent costs, implying a focus on managing throughput to match task complexity with model capabilities.
- OpenAI ChatGPT
- OpenAI is working on custom AI inference chips and integrated compute strategies, suggesting a focus on improving the throughput and efficiency of their AI systems.
- Latency
- Latency measures the time it takes for a single task to complete, while
- Bandwidth
- Bandwidth refers to the maximum rate of data transfer across a given
- Processing Power
- Processing power is the raw computational capability, whereas
The term throughput originates from industrial engineering and operations management, referring to the rate at which a system produces goods or services. It was adopted by computer science to describe system performance.
Try sending multiple, simple prompts to an AI chatbot like ChatGPT or Google Gemini in rapid succession. Observe how quickly the system responds to each prompt, especially when sending several at once. This can give you a feel for the system's ability to handle concurrent requests, a factor in its throughput.
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- PromptRole-play a student struggling with math for AI tutor simulationSimulate a student's math struggles and uncertainty for AI tutor evaluation. · 5 min
- TemplateReusable workflow templates for healthcare teamsTeams can create and share templates for common workflows like documentation or prior authorizations. · 15 min
- TemplateTemplate for building an AI tool tailored to your workA structured approach to defining and building a custom AI tool for specific business needs. · 15 min
- TemplateEvals CSV prompt list for agent skillsA CSV file template for listing prompts used in evaluating agent skills. · 20 min
- HeuristicDouble coding agent task completion with docs linksProviding coding agents with a documentation link significantly increases their task completion rate. · 2 min
Questions people ask
- What is a good throughput number?
- A 'good' throughput number depends heavily on the specific AI model, task, and hardware. For large language models, hundreds or thousands of queries per second might be considered high throughput in certain contexts.
- How does throughput affect AI costs?
- Higher throughput generally means lower cost per task, as more work is done with the same or less computational resources over time.
First seen on Subvolts Mar 13, 2026. Updated Oct 14, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.