George M.

Voice AI: Building Agents, Tool Reviews & Lessons Learned

A build-oriented reference for voice AI agents: how to choose an architecture, which platforms and frameworks are worth using in 2026, and the operational lessons that separate a good demo from a system that survives real calls. Organized as a path — architecture decision first, then platforms/frameworks, then the speech-model layer, then build craft, then the hard-won production lessons on latency, barge-in, and eval. Tag by stage to follow the path, by layer to compare tools, or by "lessons-learned" to jump straight to what breaks. Landscape moves monthly; notes flag what's dated. Researched Jul 2026.

Edition 1◆July 24, 2026◆24 entries

01 — cekura.ai

Why Voice AI Fails in Production? Building Resilient Voice AI Systems for Production

Editor's note — LESSONS LEARNED — why voice AI fails in production and how to build for it. The mindset shift: most stacks assume providers respond eventually, but production can't wait — monitor time-to-first-byte per request, enforce latency budgets, and reroute to a fallback provider before the caller hears dead air. Decouple conversational state from audio flow so a provider hiccup doesn't lose the turn. Resilience must be tested, not assumed.

#latency#lessons-learned#production#reliability#voice-ai

02 — Evalgent

Evalgent - AI Voice Agent Testing: The Complete Guide (2026)

Note — LESSONS LEARNED — the eval framework stated for a non-specialist, and one warning worth the visit alone: be careful with LLM-as-judge scoring of transcripts — it rates the words, not the call. An agent can score high on the transcript while the caller hangs up furious. Always pair model scoring with outcome metrics (task success, containment, interruption recovery). Report latency at P50/P90/P99 — the tail is the experience.

#evaluation#lessons-learned#testing#voice-ai

03 — Hamming AI

How to Evaluate Voice Agents: Complete Framework for Testing & Monitoring | Hamming AI Resources

Note — LESSONS LEARNED — how to eval before you ship and monitor after. Concrete CI/CD gates: block merges on >3% task-completion drop, >10% latency increase, >2% WER increase. Shadow-mode testing against production recordings predicts real-world impact ~95%. Turn every production failure into a permanent test case. The discipline that keeps a working agent working through prompt changes.

#evaluation#lessons-learned#production#testing#voice-ai

04 — Future AGI

Voice AI Barge-In and Turn-Taking: A 2026 Implementation Guide

Note — LESSONS LEARNED — the implementation-level companion to the Hamming runbook. Real numbers: Silero VAD as default, minimum-duration guard (200–300ms sustained voice, confidence >0.7) cuts false barge-ins 60–80%, and when barge-in fires TTS must stop within 60ms or it feels ignored. False-barge-in is a worse failure than a slightly slow one — design conservative. Keep open while tuning VAD.

#barge-in#latency#lessons-learned#turn-detection#voice-ai

05 — Hamming AI

Voice Agent Interruption Handling: Barge-In, Backchannels, and Turn Detection | Hamming AI Resources

Note — LESSONS LEARNED — the single best operational read here, drawn from analysis of 4M+ production calls across 10k+ agents. Barge-in isn't one setting: it's a policy deciding when caller speech should stop the agent, when it's just a backchannel ('uh-huh'), when it's noise to ignore, and what to log so QA can replay the call. A fast agent can still feel rude. This is the runbook for why.

#barge-in#lessons-learned#production#turn-detection#voice-ai
Voice AI Prompting Guide | Vapi
docs.vapi.ai

08 — docs.vapi.ai

Voice AI Prompting Guide | Vapi

Note — The best voice-specific prompting guide, and it applies no matter which platform you build on. Core insight text-prompt habits miss: every token costs latency because the system prompt reloads each turn, so a bloated prompt = dead air. Enforce conversational brevity in the prompt itself ('max two sentences, never list more than three options'). This is flow control implemented as prompt.

#build-guide#latency#prompting#voice-ai

09 — inworld.ai

Best Speech-to-Speech APIs in 2026: Compare Top Providers

Note — The speech-to-speech / TTS layer surveyed, including the OpenAI Realtime API's real tradeoff stated plainly: native audio-in/audio-out kills transcription latency and preserves affect, but commits you to OpenAI's models — no swapping to Claude or Gemini, no cheaper off-peak routing, and single-vendor cost that climbs at scale. Vendor-authored; the lock-in analysis is the durable part.

#review#speech-models#tts#voice-ai

10 — Coval

Best STT Providers 2026: Independent Benchmarks & How to Choose | Coval

Note — The STT layer, with independent benchmarks and — most useful — dated model releases so you can tell what's actually current. Deepgram Flux added integrated end-of-turn detection (no external VAD, saves 200–600ms); AssemblyAI Universal-3 Pro at 5.6% WER; NVIDIA Parakeet as the strongest self-host option. Also the sharpest reason to stay cascaded: if you need a transcript (compliance, audit, analytics), S2S can't give you one on the hot path.

#benchmarks#comparison#speech-models#stt#voice-ai

11 — Arun Baby

Voice Agent Frameworks: LiveKit & Pipecat

Note — The architecture explainer that makes LiveKit + Pipecat click. Don't build WebRTC from scratch — NAT traversal, codec negotiation, jitter buffers, STUN/TURN will eat months. LiveKit handles the network layer (agent joins a Room as a headless participant), Pipecat handles the VAD/STT/LLM/TTS pipeline, SIP bridges the phone network. Dated 2024 so treat specifics as stale, but the mental model holds.

#architecture#build-guide#framework#voice-ai

12 — GitHub

GitHub - pipecat-ai/pipecat: Open Source framework for voice and multimodal conversational AI

Note — Pipecat itself — the repo. Open-source Python, hit stable v1.0 April 2026. Models an agent as a pipeline of frame processors (audio in → STT → LLM → TTS → audio out), 60+ swappable service integrations, plus Flows for conversation state machines and Whisker for live pipeline debugging. Start with `pipecat init quickstart`. This is the one to prototype on if your team writes Python.

#build-guide#framework#open-source#voice-ai

13 — ThinnestAI

Pipecat vs LiveKit vs Bolna (2026): Best Open-Source Voice AI Framework

Note — The honest 'here's when each beats the one we build on' comparison of LiveKit Agents, Pipecat, Bolna, TEN. Clean mental model: LiveKit = WebRTC media server + agents layer (best if you want transport/telephony and agent logic from one source); Pipecat = pipeline of frame processors you assemble (best for control and prototyping). Disclosed LiveKit bias, fair anyway.

#comparison#framework#open-source#voice-ai

14 — TECHSY

6 Best Open-Source Voice Agent Frameworks (2026)

Note — The open-source framework ranking, with live GitHub numbers (pulled Jul 14 2026: Pipecat 13.4k stars, LiveKit Agents 11.4k, TEN 10.9k). Verdict: Pipecat for the widest integration library and most momentum, LiveKit at scale + native telephony, TEN for multimodal graphs, Bolna to get a phone agent live in an afternoon. From a team that rebuilt Pipecat→LiveKit when a client jumped 20→400 concurrent calls.

#comparison#framework#open-source#review#voice-ai

19 — TECHSY

Retell vs Vapi vs Bland: We Built on All 3 (2026)

Note — The 'we built the same agent on all three' comparison — Retell vs Vapi vs Bland, by a team that ships on them. 2026 verdict: Retell for fastest path to a managed low-latency phone agent (~600ms, HIPAA standard); Vapi for full control if you can absorb five vendor invoices; Bland for high-volume outbound with deterministic Pathways. Also notes Retell natively supports MCP as tools now.

#comparison#lessons-learned#platform#review#voice-ai

20 — The Voice AI Wiki

Voice agent frameworks: Pipecat, LiveKit Agents, and friends | The Voice AI Wiki

Note — The vendor-neutral framing of the build-vs-buy layers. Default to an open-source framework (you keep control of data, providers, logic); reach for a managed platform only when your needs are standard and you'd rather configure than operate; roll your own orchestrator only when neither fits — because then you maintain the real-time loop forever. The best one-paragraph decision rule in the set.

#comparison#framework#start-here#voice-ai

23 — inworld.ai

Cascaded vs Speech-to-Speech Voice Architecture - Inworld AI

Note — The counterweight to the 'cascaded always wins' view. Makes the real case for speech-to-speech: one model that hears tone, hesitation, sarcasm — everything a transcript throws away — and a component-level latency budget showing a well-tuned cascade can still beat some S2S models voice-to-voice. Vendor-authored (Inworld sells inference), so read the tradeoff matrix, not the conclusion.

#architecture#comparison#latency#voice-ai

24 — Gradium

Cascaded Voice Agents vs Speech-to-Speech: Architecture Tradeoffs in 2026

Note — START HERE if you're choosing an architecture. The clearest statement of why cascaded (STT→LLM→TTS) still dominates production in 2026: modularity. You can swap Claude for GPT or upgrade your STT without retraining anything, and LLMs move too fast to lock into a single speech-to-speech vendor. Read before you pick a platform, because the platform partly picks the architecture for you.

#architecture#comparison#start-here#voice-ai

Save this collection

Keep it in your own Collections list — it stays in sync with the latest edition.

Comments

Sign in to join the conversation.