OpenAI Advanced Voice is ChatGPT's built-in real-time voice dialogue capability. Based on the GPT-Realtime model, it implements end-to-end voice input and output, emotion perception, interruption interaction, and multiple rounds of natural dialogue. Different from the traditional speech-to-text → LLM → text-to-speech pipeline, Advanced Voice adopts a native voice multi-modal architecture, with latency as low as 200-300ms, and supports a variety of preset timbres, custom instructions and real-time translation scenarios.

OpenAIAdvancedVoice

Core parameters and statistics

OpenAI Advanced Voice is ChatGPT’s built-in real-time voice dialogue capability and is also the main delivery form of OpenAI Realtime API on the consumer side. It is not just a voice chat function, but integrates voice input, semantic understanding, emotion perception, and voice output into an end-to-end multi-modal interaction system. Officially positioned as "natural, real-time conversations with ChatGPT", the goal is to allow users to communicate with AI as naturally as they do with real people.

Projects	Public Information
Official positioning	Real-time voice dialogue mode, supporting natural interruptions and emotional expression
Underlying model	GPT-Realtime series (latest gpt-realtime-2.1)
Interaction mode	End-to-end voice ↔ voice, supports text fallback
Latency indicator	About 200-800ms (end-to-end, depending on the network and load)
Number of sounds	About 10+ preset sounds (Alloy, Echo, Shimmer, Coral, Ember, Fable, Nova, Sage, etc.)
Supported languages	Speech recognition and generation in 50+ languages, including Chinese, English, Japanese, Korean, Spanish, French, German, etc.
Multi-modal capabilities	Some versions support camera visual input (understand the screen and describe it with voice)
Deployment platform	iOS / Android native App, Web, desktop
API available	Open via Realtime API (WebSocket / WebRTC)
Business model	ChatGPT subscription (Plus/Pro/Team/Enterprise) + API pay-as-you-go

Interaction delay: Advanced Voice's real end-to-end delay fluctuates between 200ms~800ms, which is much faster than the traditional three-stage pipeline of "speech to text→LLM→text to speech" (usually 1.5~3s). Experience is highly dependent on network quality, server load, and Voice Activity Detection (VAD) threshold settings.

Multi-modal expansion: Starting in the second half of 2025, Advanced Voice will gradually integrate camera vision capabilities - users can point the camera at objects, and AI will describe what they see in real-time voice. This makes it evolve from a "voice chat tool" to a "real-time multi-modal assistant".

API Parity: Advanced Voice in ChatGPT shares the underlying model with the OpenAI Realtime API. The API side allows developers to customize VAD parameters, sounds, system commands and tool calls, while the consumer side version prioritizes ensuring a universal experience and safety guardrails.

User and market recognition

Advanced Voice Mode quickly became one of ChatGPT’s most talked-about features after its launch, but OpenAI does not disclose independent MAU or enterprise customer volume data.

Consumer Penetration: OpenAI announced that ChatGPT’s weekly activity exceeded 400 million at the end of 2025, among which Advanced Voice is one of the most commonly used features by Plus/Pro users. A large number of users on social media (Reddit, X, Xiaohongshu) share voice interaction cases, indicating that it has formed a self-propagation effect.

Developer Market: Since its public preview in December 2024, the Realtime API has gained significant developer adoption in the following scenarios:

• Voice Agent: A voice robot for customer service, reservation, and education and coaching, replacing the traditional IVR system.
• Real-time Translation: Use Realtime Translation API to build multi-lingual simultaneous interpretation applications.
• Wearable devices: embedded scenarios such as smart glasses, headphones, and car voice assistants.

Industry benchmarking: Compared with Google Gemini Live (real-time voice dialogue) and Grok Voice (X platform voice mode), Advanced Voice is in the first echelon in terms of natural response, emotional expression, and smoothness of interruption processing. However, Gemini Live has deeper system integration on the Android side, and Grok Voice has more advantages in X platform context continuity.

Cost advantage

• C-side/Individual: Usually a free version is provided to experience the core functions, and high-frequency use requires a paid package subscription.
• API/Developer: Billed by call volume, suitable for development teams that can be flexibly integrated into their own systems.
• Enterprise/Privatization: Contact the business owner to obtain customized quotation and deployment plan. The specific price is subject to the official real-time pricing page.

Main functions

Advanced Voice is not a single function, but a complete real-time voice interaction system. The following are core capabilities and their synergies:

• End-to-end voice dialogue: Without the need for intermediate transliteration steps, the model directly understands the tone, pauses, speed and emotion in the voice, and generates emotionally colored voice responses. Implementation Tips: It is recommended to test the speech recognition accuracy in a noisy environment. The performance of the model may decrease when the signal-to-noise ratio is lower than 15dB.
• Natural Interruption and Resume: Users can interrupt at any time while the AI ​​is speaking, and the model can immediately stop, understand the new input, and continue the conversation. Expert View: This capability seems simple, but in fact it requires in-depth collaboration between VAD and inference management - the model needs to quickly determine whether the user is interrupting (should stop replying) or background noise (should be ignored). If the threshold is set too tight, the experience will be reduced, and if it is too loose, tokens will be wasted.
• Emotion Perception and Expression: The model can identify the emotions in the user's tone (excitement, confusion, frustration, sarcasm, etc.) and match the corresponding tone when replying. Acceptance focus: The current model still has a gap in accuracy of emotion recognition for non-English languages, and emotional misjudgment may occur in some contexts such as Chinese and Japanese.
• Multi-lingual mixed conversation: Supports the use of multiple languages ​​in the same conversation. For example, the user asks a question in Chinese and the AI ​​replies in English. Suitable for cross-language communication scenarios.
• Visual understanding (some versions): In camera mode, AI can analyze the screen content in real time and describe it with voice. Expert View: This is the key function that Advanced Voice takes from "voice assistant" to "multi-modal real-time assistant". The current visual understanding delay is about 200-400ms higher than that of pure speech scenes, and the recognition accuracy in low light and environment will decrease.
• Customized commands and memory: Supports calling ChatGPT's custom commands and memory functions in voice conversations, making the voice assistant's reply style and knowledge background closer to personal preferences.
• Tone selection: Provides a variety of preset timbres with different timbres

There are differences in emotional expressiveness - for example, "Coral" is more lively and natural, "Alloy" is more neutral and professional, and "Nova" is more warm and approachable.

Hidden linkage: These functions are not isolated from each other. For example, "end-to-end speech" provides a low-latency basis for "interruption", while "emotion perception" relies on the natural expression ability of multi-modal input. When the user interrupts and changes the tone, the model needs to handle the three tasks of audio interruption, semantic switching and emotional matching at the same time, which is impossible for the traditional three-stage pipeline.

Model and version evolution

Advanced Voice version evolution is closely synchronized with OpenAI Realtime API model versions. Since the functional release of the ChatGPT client does not exactly correspond to the API model version, the following is organized according to the minimum verifiable milestones.

Realtime API model context

Model version	Release time	Key changes
gpt-4o-audio-preview-2024-10-01	2024-10	The first native audio Chat Completions model, supporting audio ↔ text
gpt-4o-realtime-preview-2024-10-01	2024-10	The first WebSocket real-time voice API, supporting VAD and low-latency streaming
gpt-4o-realtime-preview-2024-12-17	2024-12	Improve VAD accuracy, reduce hallucinations, add WebRTC support
gpt-realtime-2.0	~2026-03	GPT-Realtime official version, delay significantly reduced, voice clarity improved
gpt-realtime-2.1	~2026-06	The latest, improved multilingual accuracy, emotion control, and more stable low latency

ChatGPT Advanced Voice feature milestones

• 2024-09: OpenAI has limitedly opened Advanced Voice Alpha to Plus users on the ChatGPT mobile terminal. Initially, only 5 preset sounds are supported.
• 2024-12: Advanced Voice is open to all Plus and Pro users, adding custom command support and expanding the sound library to 9 types.
• 2025-04: Desktop Advanced Voice is online, supporting real-time voice conversations in web browsers.
• 2025-09: Introducing camera visual understanding capabilities (tested by some users), Advanced Voice is upgraded from pure voice to a multi-modal real-time assistant.
• 2026-02: Advanced Voice fully integrates the GPT-Realtime 2.0 model, reducing latency by approximately 30%.
• 2026-05: Realtime Translation API is released, Advanced Voice gains multi-lingual real-time translation capabilities.

Release Note: The Advanced Voice feature version of the ChatGPT client does not correspond one-to-one with the API model version. OpenAI usually releases new models through the API first, and then gradually pushes them to the ChatGPT application. Therefore, there may be a time difference between the functional changes perceived by users in actual use and the API changelog.

Technical advantages

Advanced Voice's technical advantages are rooted in "native multi-modal architecture" rather than simple assembly of pipelines. The following is expanded from the three dimensions of mechanism → effect → scene.

Native voice multi-modal architecture

Mechanism: Traditional speech AI uses a three-stage pipeline of "ASR→LLM→TTS" - speech is first converted to text, the text model generates a reply, and then the speech is synthesized. Each cross-modal transition introduces delay (usually 500ms-1s/segment) and loses paralinguistic information such as tone, rhythm, emotion, etc. Advanced Voice is based on the native multi-modal design of the GPT-Realtime model. It processes audio tokens directly within the model and can generate voice responses without intermediate text transcription.

Effect: The end-to-end delay is reduced to 200-800ms (1/3~1/5 of the three-stage pipeline), and the emotion, speech speed, and pauses in the voice are completely preserved, and the naturalness of the dialogue is significantly improved.

Applicable Scenarios: Voice Agents, emotional companionship, and intelligent customer service that require high naturalness of interaction - in these scenarios, users have extremely low tolerance for "robot feeling", and the smoothness brought by native multi-modality is the key to differentiation.

Voice activity detection (VAD) and interruption management

Mechanism: Realtime API has built-in server-side VAD, which can determine whether the user is speaking in real time by analyzing the energy, frequency and voice pattern of the audio stream. When VAD detects an interruption in user intent, the model interrupts the current output, clears the inference cache, and immediately processes the new input.

Effect: Users can insert new questions without waiting for the AI ​​to finish speaking, and the conversation rhythm is closer to that of real people. However, the VAD threshold needs to be balanced - too tight will cause the background sound to trigger interruptions by mistake, while too loose will cause the user to have to speak louder to interrupt.

Applicable Scenarios: Any conversation scenario that requires rapid alternation of speech (such as debates, brainstorming, customer service calls). However, in extremely quiet or noisy situations, it is recommended to manually adjust the VAD threshold through API parameters.

Speech emotion recognition and synthesis

Mechanism: The model retains paralinguistic features (pitch, loudness, speaking speed, timbre formants) in the audio token, can perceive the user's emotional state during inference, and match the corresponding emotional expression when generating speech.

Effect: AI can express emotional states such as "empathy," "confusion," and "excitement" through tone of voice instead of a uniform and steady tone. This directly contributes to trust building and user stickiness.

Applicable scenarios: Psychological counseling, language learning, business negotiation exercises and other scenarios that are sensitive to emotional expression. It should be noted that the model still has understanding bias in complex emotional expressions such as sarcasm and irony.

WebRTC and WebSocket dual channels

Mechanism: Realtime API supports both WebSocket (low-level audio stream control) and WebRTC (browser native real-time communication) connection methods. WebRTC uses the ICE/STUN/TURN protocol stack to automatically handle NAT penetration, bandwidth adaptation, and audio encoding and decoding.

Effect: The end-to-end delay of the WebRTC path is further reduced by about 15-30% than the WebSocket path, which is especially suitable for browser-side and mobile-side voice applications. Developers do not need to implement audio codec and network negotiation by themselves.

Applicable scenarios: Browser voice agent, mobile real-time translation, wearable devices (such as smart glasses, headphones) - these scenarios have extremely high requirements on latency and connection stability.

Function Calling integration

Mechanism: Realtime API supports triggering function calling in voice conversations. The model parses the user's intention in the audio stream and calls external tools (query the database, create work orders, place orders, etc.), and the results are returned to the user through voice.

Effect: Voice Agent is no longer limited to information Q&A and can directly perform business operations. However, the reliability of tool calls is still lower than that of plain text mode - voice input is more ambiguous, and it is recommended to set voice confirmation ("Are you sure you want to place an order?") for key operations.

Applicable scenarios: Voice ordering, reservation system, knowledge base query and other scenarios that require voice interaction and business system linkage.

How to use

The usage entrance of OpenAI Advanced Voice is divided into consumer side (ChatGPT application) and developer side (Realtime API), and the respective access methods are significantly different.

ChatGPT consumer side

Platform	Access	Prerequisites
iOS / Android	ChatGPT App → Headphone icon in the lower right corner → Click "Advanced Voice"	Plus/Pro/Team/Enterprise Subscription
Web (desktop)	chatgpt.com → Headphone icon near the input box → Turn on voice	Plus/Pro/Team/Enterprise subscription
macOS / Windows Desktop	ChatGPT Desktop App → Shortcut Key/Voice Button	Plus/Pro/Team/Enterprise Subscription

Usage steps:

1. Make sure you subscribe to ChatGPT Plus ($20/month) or higher.
2. Open ChatGPT App or Web, click the "Advanced Voice" mode entrance next to the voice input button.
3. Select a preset tone (Alloy, Echo, Shimmer, Coral, Ember, Fable, Nova, Sage, etc.).
4. Start a natural voice conversation - you can speak directly and the AI ​​will respond automatically; you can also interrupt the AI ​​to adjust the problem at any time.
5. To end the conversation, click the Close button.

Developer API

Access method: Use WebSocket or WebRTC connection through OpenAI Realtime API.

Prerequisites:

• A valid OpenAI API Key.
• Activate the billing and quota of Realtime API.
• Backend services support WebSocket or WebRTC signaling.

Python quick example (WebSocket way):

import asyncio
import websockets
import json

API_KEY = "<YOUR_API_KEY>"
MODEL = "gpt-realtime-2.1"

async def realtime_session():
    url = f"wss://api.openai.com/v1/realtime、model={MODEL}"
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "OpenAI-Beta": "realtime=v1"
    }
    async with websockets.connect(url, extra_headers=headers) as ws:
        # Configure session
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "modalities": ["audio", "text"],
                "instructions": "You are a friendly voice assistant. Please reply in Chinese.",
                "voice": "coral",
                "input_audio_format": "pcm16",
                "output_audio_format": "pcm16",
                "input_audio_transcription": {"model": "gpt-4o-transcribe"},
                "turn_detection": {
                    "type": "server_vad",
                    "threshold": 0.5,
                    "prefix_padding_ms": 300,
                    "silence_duration_ms": 600
                }
            }
        }))
        # Subsequent processing of audio streams, events, etc.
        # See OpenAI Realtime API documentation for details

asyncio.run(realtime_session())

JavaScript quick example (WebRTC browser mode):

import { RealtimeAgent, RealtimeSession } from "@openai/agents/realtime";

const agent = new RealtimeAgent({
  name: "Assistant",
  instructions: "You are a friendly voice assistant.",
});

const session = new RealtimeSession(agent, {
  model: "gpt-realtime-2.1",
});

await session.connect({
  apiKey: "ek_<ephemeral_key_from_server>",
});

Phased implementation suggestions

1. Pilot Phase (1-2 weeks): Select 1-2 high-value scenarios (such as first-round customer service screening, voice note transcription), and manually test using Advanced Voice on ChatGPT Plus to verify response accuracy and user acceptance.
2. Contrast phase (2-4 weeks): Build a prototype through API and conduct A/B testing with the existing pipeline (ASR→LLM→TTS), focusing on comparing latency, user satisfaction, and task completion rate.
3. Expansion Phase: After verifying the ROI, gradually increase the voice agent coverage and set up human-in-the-loop for key business operations.

Product Pricing

The pricing model is subject to the official real-time page. Usually a freemium or subscription system is used, and basic functions can be used for free. Advanced functions or high-frequency use require paid subscriptions, and users are advised to evaluate the optimal solution based on actual usage.

Application scenarios

The implementation scenarios of Advanced Voice span across consumer and commercial levels. The following are three typical scenarios.

Voice intelligent assistant and life companion

• Task Type: Daily Q&A, information inquiry, schedule management, chatting and companionship.
• Actual benefits: In scenarios where hands are busy such as driving, cooking, and sports, voice interaction is 3-5 times more efficient than typing. In the emotional companionship scenario, Advanced Voice's emotional expression retention rate is about 40% higher than that of traditional TTS (deduced value, unofficial promise).
• Verification Points: Test the speech recognition accuracy in moderately noisy environments (such as cars, coffee shops); evaluate the model's context-preserving ability in long conversations (>30 minutes).

Enterprise customer service and reservation system

• Task Type: First-round customer screening FAQ self-service answer, appointment confirmation, order inquiry.
• Actual benefits: Compared with traditional IVR menu-based interaction, Advanced Voice's "just speak" experience can increase the user's self-service resolution rate from 60% to 80-85% (deduction value); the number of sessions that a single customer service person can handle simultaneously increases from 1 to 3-5 (AI-assisted mode).
• Verification Points: It is necessary to focus on testing the recognition accuracy of professional terms (such as insurance terms, medical terms); key operations (payment, order cancellation) must be set up with a double guarantee of voice confirmation + SMS secondary confirmation.

Language learning and cross-language communication

• Task type: foreign language speaking practice, real-time translation, cross-language business communication.
• Actual benefits: Realtime Translation API supports real-time translation in 50+ languages, with an end-to-end delay of about 500ms-1.5s, which basically meets the available standards for simultaneous interpretation. Language learners can get instant pronunciation and grammar feedback during voice conversations with AI.
• Verification Points: In translation scenarios, the translation accuracy of professional terms needs to be compared; in oral practice scenarios, there is still a gap between the accuracy of feedback and the teacher's level, and it is not suitable for the refined pronunciation correction needs of advanced level learners.

Applicable people

Individual users

• ChatGPT Subscribers (Plus/Pro): Daily voice Q&A, life assistant, foreign language learning. It is not suitable to be used for voice consultation in professional fields such as medical diagnosis and legal consultation - Advanced Voice is not an expert system, and its answer accuracy is limited by the knowledge boundaries of the underlying model.
• Multi-tasking scenario users: Scenarios where hands/eyesight are occupied such as driving, cooking, fitness, etc. The value of Advanced Voice is most evident here.

Developers and Product Team

• Voice product developer: Use Realtime API to build voice agents, real-time translation, and wearable device voice interaction. It is necessary to pay attention to the linear growth of API costs after scale-the price of Audio token is 4-10 times that of text token. In high concurrency scenarios, the cost needs to be estimated in advance.
• Enterprise IT and AI Team: Integrate Advanced Voice or Realtime API into customer service systems, appointment processes, and internal knowledge base voice queries. Not suitable for scenarios that require privatized deployment (OpenAI does not currently provide privatized deployment options for Realtime API).

Corporate Buyer

• Customer Service/Operations Leader: Evaluate the ROI of Advanced Voice replacing or supplementing human agents. It is necessary to pay attention to the misjudgment rate of the voice agent, changes in user satisfaction and the integrity of the compliance audit chain.
• Compliance and Security Leader: Evaluate voice data processing and retention strategies. OpenAI offers a zero data retention option for the API, but voice data from ChatGPT consumers may be used for model improvement (except for Enterprise subscriptions).

Not suitable for the crowd:

• Organizations that require a fully offline/private voice assistant.
• Real-time control scenarios that require voice interaction delay <100ms (such as voice control of industrial equipment).
• For voice interaction in high-noise environments (>85dB), the accuracy of VAD and ASR will significantly decrease.

Summary and Outlook

OpenAI Advanced Voice is one of the most mature end-to-end real-time voice dialogue systems on the market. Its core advantage lies in "native multi-modality" rather than "pipeline assembly" - which makes it significantly better than the traditional ASR→LLM→TTS solution in terms of interaction naturalness, latency and emotional expression.

From the perspective of technical architecture, the GPT-Realtime model's native audio token processing mechanism, server-side VAD interruption management WebRTC/WebSocket dual-channel connection constitute three barriers to differentiation. From a commercial perspective, the dual-track model of consumer subscription + API pay-as-you-go allows it to cover both individual users and commercial developers.

Current Limitations:

1. Cost Barrier: The price of audio token is 4-10 times that of text. The cost of API for large-scale commercial deployment needs to be accurately estimated.
2. Not available offline: Advanced Voice relies entirely on cloud inference, has no offline mode, and is limited to sensitive network scenarios.
3. Non-English Language Gap: The quality of emotion recognition and speech generation in non-English languages ​​such as Chinese, Japanese, and Arabic is still lower than English.
4. Lack of privatization: There is no privatized deployment solution for Realtime API, which poses an obstacle to strong compliance industries such as finance and medical care.
5. Audio Security: Injection attacks on voice interactions (such as manipulating model behavior through adversarial audio) are an emerging AI security challenge.

Procurement/Adoption Risk Assessment:

• Pilot priority: It is recommended to start with 1-2 low-risk scenarios (such as FAQ voice self-service, voice notes), use ChatGPT Plus or API for small-amount verification, and confirm response quality and user acceptance before expanding.
• Cost Modeling: Before scaling up, perform cost modeling based on the estimated average daily conversation volume, average conversation duration, and Audio token proportion to avoid monthly API bills that significantly exceed expectations.
• Compliance Traces: Complete logs and audits of the voice agent's conversation content to ensure that there are well-documented records in the event of misjudgment or compliance disputes.
• Key Contract Clauses: When an enterprise purchases a ChatGPT Enterprise or API enterprise contract, it must focus on confirming whether the audio data is used for model training, service availability SLA, and the definition of OpenAI's legal liability for the behavior of the voice agent.

Related tools: ElevenLabs, udio

Version Info

• GPT-4o Realtime Preview ：OpenAI's first public real-time speech API preview version supports WebSocket streaming speech input and output, and provides back-end model capabilities for Advanced Voice Mode. There is no official precise date yet. （2024-12-17）
• GPT-4o Realtime Early Preview ：OpenAI's early real-time voice API test version opens WebSocket real-time audio conversation capabilities to some developers, supporting function calling and multi-round conversations. There is no official precise date yet. （2024-10-01）
• GPT-4o Audio Preview ：OpenAI's first API model to support native audio input and output, which can process audio data through the Chat Completions endpoint, the predecessor technology verification version of Advanced Voice Mode. There is no official precise date yet. （2024-10-01）
• GPT-Realtime 2.1 (Advanced Voice) ：The latest real-time speech model version supports lower-latency speech-to-speech interaction, improved voice activity detection (VAD), enhanced multi-language recognition capabilities, and richer emotional expression control. Deep integration with ChatGPT mobile and desktop Advanced Voice modes. （~2026-06）
• GPT-Realtime 2.0 ：The first official version of the GPT-Realtime series, compared with the preview version, it significantly reduces latency, improves voice clarity, and introduces WebRTC connection support. There is no official precise date yet. （~2026-03）