Speech-to-speech
Also written: S2S, Speech to Speech APIs, Speech-to-speech models
AI that directly converts spoken audio input into spoken audio output, skipping text.
Speech-to-speech AI lets you talk to a computer, and it talks back to you in spoken words, without you needing to type anything.
Think of it like having a conversation. You speak, the AI understands your voice and responds with its own voice. This is different from older systems where the AI would convert your speech to text, process that text, and then convert the text back to speech.
Speech-to-speech (S2S) is a modality where spoken audio input is directly transformed into spoken audio output. Unlike traditional cascaded systems that rely on separate Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) models, direct S2S models aim to perform the entire conversion in a single, end-to-end neural network. This can reduce latency and potentially improve expressiveness by preserving nuances from the source speech that might be lost in text representation.
Recent advancements in end-to-end neural networks allow for lower latency and more natural-sounding voice interactions. Companies are integrating these capabilities into developer tools and products, enabling real-time voice agents and audio content creation.
- Gemini Live audioGoogle · Simon Willison ·
- Builder Bootcamp: Realtime - EventOpenAI · OpenAI Academy ·
- Audio blog postsOpenAI · OpenAI Developers Blog ·
- Developer notes on the Realtime APIOpenAI · OpenAI Developers Blog ·
- Voice Agent observability with LangSmith 🌟Google · Google for Developers on YouTube ·
- Custom VoicesxAI · xAI Docs: Guides ·
- Introducing the Realtime APIOpenAI · OpenAI News ·
- Google Gemini
- Google Gemini's Live feature supports building voice agents that can interact using speech-to-speech capabilities.
- OpenAI ChatGPT
- OpenAI's Realtime API enables low-latency speech-to-speech AI conversations, and their audio tools allow for custom voice generation.
- Cascaded ASR-TTS
- This involves separate speech recognition and speech synthesis
- Voice Assistants
- While voice assistants often use speech-to-speech, the term it
The concept of speech-to-speech translation has roots in early speech processing research, with direct end-to-end models gaining significant traction in recent years due to deep learning advancements.
Use a voice assistant like Google Assistant or Siri. Speak a command or question, and listen to its spoken response. This demonstrates a common application of speech-to-speech technology in everyday devices.
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- ToolxAI Custom Voices API for Text to SpeechClone and use custom voices in xAI's TTS and Speech to Speech APIs. · 5 min
- PromptRole-play a student struggling with math for AI tutor simulationSimulate a student's math struggles and uncertainty for AI tutor evaluation. · 5 min
- TemplateReusable workflow templates for healthcare teamsTeams can create and share templates for common workflows like documentation or prior authorizations. · 15 min
- TemplateTemplate for building an AI tool tailored to your workA structured approach to defining and building a custom AI tool for specific business needs. · 15 min
- TemplateEvals CSV prompt list for agent skillsA CSV file template for listing prompts used in evaluating agent skills. · 20 min
Questions people ask
- Does speech-to-speech always skip the text step?
- Not always. Some systems use speech-to-speech as an umbrella term, but true direct speech-to-speech models aim to bypass intermediate text transcription for lower latency and better expressiveness.
- What are the benefits of direct speech-to-speech?
- Direct speech-to-speech can offer lower latency, potentially more natural-sounding output, and the ability to preserve vocal characteristics like tone and emotion from the source audio.
First seen on Subvolts Oct 1, 2024. Updated Oct 12, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.