Voice AI infrastructure: The 2026 stack explained

This guide describes the layers of a production voice AI stack—speech recognition, turn-taking, language models, orchestration, tools and MCP, knowledge and memory, speech synthesis, telephony, and guardrails, testing, and monitoring—and explains how ElevenAgents implements each, using Eleven v4 as the example voice layer. It covers two build modes (Custom and Realtime), Scribe v2 Realtime transcription at roughly 150 ms, a proprietary turn-taking model, workflow orchestration nodes, MCP tool approval modes, RAG latency reduced from 326 ms to 155 ms, and Eleven v4 Turbo figures of about 100 ms inference latency and about 150 ms time to first speech. It also details Guardrails 2.0, simulation and tool-call tests, A/B experiments, and observability metrics.