How OpenAI delivers low-latency voice AI at scale
By Yi Zhang and William McDonald, Members of Technical Staff
TL;DR
- OpenAI rearchitected its WebRTC stack to achieve low-latency voice AI at scale for over 900 million weekly users.
- The new architecture uses a 'split relay plus transceiver' model, separating packet routing from protocol termination.
- This design addresses infrastructure constraints like one-port-per-session limitations and the need for stable session ownership.
- The relay acts as a lightweight UDP forwarding layer, while the transceiver handles the stateful WebRTC session termination.
- Encoding routing metadata (ICE ufrag) into the protocol allows for deterministic first-packet routing.
- Global Relay deploys geographically distributed ingress points to shorten the initial client-to-OpenAI hop, reducing latency.
- The solution preserves standard WebRTC behavior for clients, ensuring interoperability with browsers and mobile apps.
- This architecture allows WebRTC media to run efficiently in Kubernetes without exposing thousands of UDP ports.