Top news
Investing in the public benefit OfficialAnthropic NewsOct 18Popular Musk Pledges SpaceX Shares to ‘Every Kid in America’ WebBloomberg TechnologyOct 19Popular Websites are closing their doors to AI agents WebThe Verge AIOct 17 This is the new ChatGPT Voice, powered by GPT Live OfficialOpenAI on YouTubeOct 18 How AI giants will reshape global philanthropy WebAxios AIOct 19 Gemini 4 Argon Really Surprised Me WebTwo Minute PapersOct 19 Google Gemini 4 agents just beat Minecraft with no human intervention WebEngadget AIOct 19 Who could it be? 🧐 OfficialGoogle on YouTubeOct 18 Anthropic gives outside AI watchdog “employee-like” access WebArs Technica AIOct 18 What are self-replicating ‘AI worms’, and why is Anthropic worried about them? WebThe Guardian AIOct 18
GlossaryInferenceNew to AI

Latency

Also written: Low latency, Thinking Time, Inference bottlenecks, Conversational latency, Low-latency streaming, low-latency data streaming, Inference latency, Token latency

The delay between a user's input and the AI system's response.

In plain terms

Latency is the time it takes for a system to react to your command or question. Think of it like asking a question and waiting for an answer. If the latency is low, the answer comes back quickly. If the latency is high, you have to wait longer.

For example, when you use a chatbot, latency is the time between you typing your message and the chatbot starting to reply. Lower latency makes the interaction feel more natural and immediate, like a real conversation.

The expert version

Latency refers to the time delay between a request being sent to an AI model and the first part of the response being received. This is a critical performance metric, especially for real-time applications like conversational AI, live translation, or interactive content generation. High latency can degrade user experience, making interactions feel sluggish and unnatural.

Factors influencing latency include model size, computational complexity, network conditions, and the efficiency of the inference hardware and software stack. Optimizing for low latency often involves trade-offs with model accuracy, throughput, or cost.

Why it matters now

Recent announcements from Google Gemini highlight faster and cheaper AI models, specifically mentioning "Flash" and "Flash-Lite" versions. This focus on speed and efficiency suggests a push towards reducing the time it takes for AI models to process information and generate responses, making advanced AI more accessible for high-volume applications.

Where it shows up 145 items, newest first
How each company uses it
Anthropic Claude
Anthropic's models are evaluated for their response speed, impacting user experience in conversational AI.
Google Gemini
Google Gemini is actively developing and releasing faster versions of its models, like Gemini Flash, to reduce processing time and cost for applications.
Microsoft Copilot
Microsoft Copilot aims to provide quick and responsive AI assistance, with latency being a key factor in its real-time performance.
OpenAI ChatGPT
OpenAI's models are assessed for their inference latency, a critical metric for delivering timely responses in applications like ChatGPT.
Related terms

ThroughputInferenceResponse TimeReal-time AIAPIPerformance

Often confused with
Throughput
Throughput measures how many requests an AI can handle in a given ti
Bandwidth
Bandwidth is the amount of data that can be transferred per unit of t
Processing Time
Processing time is part of latency, referring specifically to t
Origin

The term latency originates from physics and engineering, referring to a delay between cause and effect. In computing, it has long been a key metric for network and system performance.

Try this about 5 minutes

Use a chatbot like ChatGPT or Google Gemini. Ask it a simple question and notice how quickly it starts to respond. Then, try asking a more complex question or requesting an image generation and observe if the response time changes. This gives a direct feel for the latency involved in different tasks.

Try it: Playbook entries that use thisPlaybook →

The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.

Questions people ask

Is low latency always good?
Low latency is generally good for user experience, making interactions feel faster. However, achieving very low latency can sometimes come at the cost of accuracy or require more expensive hardware.
How is latency measured?
Latency is typically measured in milliseconds (ms) and represents the time from sending a request to receiving the first byte of the response.

First seen on Subvolts Dec 14, 2021. Updated Oct 17, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.