Delivering Low-Latency Voice AI at Scale: OpenAI's Infrastructure Secrets
OpenAI details the massive re-architecture of its WebRTC stack to deliver near-instant voice interactions to over 900 million users, making network latency feel invisible.
The Speed of Speech
For Voice AI to feel natural, it must operate at the speed of human speech. OpenAI, serving over 900 million weekly active users, recently detailed how they achieved this by rearchitecting their WebRTC stack to eliminate awkward pauses and clipped interruptions.
- Split Relay + Transceiver Architecture: To solve the mismatch between WebRTC's stateful nature and Kubernetes' elasticity, OpenAI separated packet forwarding (Relay) from session state management (Transceiver).
- Routing on ICE Credentials: By using the "ufrag" present in the very first packet, they created a routing hook that eliminates expensive hot-path lookups.
- Global Geo-Steering: Traffic is routed to the nearest entry point while anchoring sessions to a single transceiver, minimizing round-trip time (RTT).
The Shift Toward Proactive Defense
OpenAI’s approach represents a fundamental transition from standard cloud deployment to deep infrastructure ownership. By customizing Linux socket options like SO_REUSEPORT and implementing thread pinning (LockOSThread), they've moved beyond treating the network as a "black box." This proactive engineering allows for partial inference—starting the model's reasoning while the user is still speaking—turning the system from a "push-to-talk" machine into a true conversational partner. This level of optimization effectively builds a technical moat that is incredibly difficult for competitors relying on off-the-shelf media servers to bridge.
Source: OpenAI Infrastructure Blog