Static

OpenAI still has a WebRTC Problem

First reported by Moq.dev ·

The signal ●○○○ Compiled by AI from Moq.dev and Hacker News
Why you might care

The core protocol for real-time audio in browsers is not suitable for AI, forcing AI developers to seek alternative solutions.

What happened

OpenAI's voice AI product faces a significant challenge with its reliance on WebRTC, a technology optimized for real-time human conversation rather than AI processing. WebRTC aggressively drops late audio packets to maintain low latency, which is detrimental for AI applications that can benefit from delayed retransmissions to ensure complete context is received. This leads to AI assistants failing to understand user input, forcing users to repeat themselves. The article highlights that while AI can process audio faster than real-time, WebRTC's design, prioritizing near-zero latency for human interaction, hinders this capability. This limitation prevents AI agents from effectively catching up on lost or delayed packets, impacting the user experience and potentially wasting processing tokens. The author proposes Media over QUIC (MoQ) as an alternative that supports retransmissions and timestamp-based playback, enabling AI to handle network fluctuations more robustly.

What it means

The fundamental issue lies in WebRTC's design philosophy, which prioritizes minimizing latency for human-to-human communication by discarding late packets. This approach clashes directly with the needs of voice AI, where delayed packets can be retransmitted and processed by AI agents that operate at speeds far exceeding human perception. While WebRTC's jitter buffer is tuned for immediate playback, AI applications could leverage buffered audio, even with retransmissions, without the end-user noticing any significant delay. This suggests that the current web standard for real-time communication is a bottleneck for the advancement of more sophisticated and reliable voice AI interactions.

Furthermore, WebRTC's handling of audio playback is based on packet arrival rather than explicit timestamps, compounding issues for AI-generated speech (TTS). This requires AI systems to artificially delay transmissions, a suboptimal workaround that also fails when packet loss occurs, as WebRTC does not retransmit these late packets. Technologies like MoQ, which support timestamped playback and extended retransmission buffering, offer a more robust solution by allowing AI to manage audio streams efficiently, ensuring context integrity and a smoother user experience. OpenAI's continued use of WebRTC indicates a significant architectural hurdle that may require substantial re-engineering to overcome.

AI-written summary. May contain errors.