Reducing latency in voice AI applications comes down to three things: choosing a fast, latency-prioritized model, sending inference requests eagerly while the user is still speaking, and leaning hard into prefix caching so the model isn't starting from scratch on every turn. Get all three right, and your voice AI can respond with a visual update within one second — which is exactly the threshold where the experience goes from frustrating to delightful. Here's what building this in the real world actually looks like.
What Is the Voice In, Visuals Out AI Approach?
The idea comes from a framing by Andrej Karpathy: voice is the human-preferred input for AI, while visuals are the preferred output. That might sound obvious once you hear it, but it's not how most AI products are built today. Most AI interfaces still ask you to type a question and read a markdown-formatted text wall in return.
The voice-in, visuals-out model flips that. You speak naturally — just like you would to a colleague — and the AI responds by rendering something on screen: a visualization, an interactive control, a generated image, a filed issue, or an updated dashboard. About a third of the human brain is dedicated to processing visual information, so this output format maps directly to how we actually prefer to receive information.
Recent breakthroughs in model capability have made this genuinely feasible. Modern LLMs can now generate rich HTML, call tools, produce interactive UI components, and render illustrations — all in response to a spoken prompt. The ceiling on what AI can visually output has lifted dramatically. The harder problem, it turns out, is the input side: making voice feel seamless rather than awkward.
Why Is Voice the Preferred Human Input for AI?
Speaking is higher bandwidth than typing in almost every dimension. You produce more words per minute when you talk, but more importantly, you convey more information per word. The difference between saying okay with flat affect versus okay with rising intonation carries meaning that no text message can replicate. That's why, when something genuinely important needs to be communicated, people jump on a call instead of firing off a Slack message.
The problem is that most people's experience with voice AI has been deeply underwhelming — think Siri failing to turn the lights on, or ChatGPT voice mode getting confused and awkward. The models have been simultaneously slow and not very smart, which is a rough combination. That has made a lot of developers skeptical of voice as an input modality. But the technology has moved fast, and the right architecture can make voice feel completely natural.
What Are the Human Latency Limits That Actually Matter?
We've known since the 1960s that for a computer interface to feel truly instant, it needs to respond within about 100 milliseconds — one tenth of a second. That's the gold standard. In practice, network latency and computation mean we often flex up from there. The practical ceiling for a standard interaction is around 1,000 milliseconds — one full second — before users start losing their train of thought or mentally moving on.
For a fully conversational voice-in, voice-out experience, the threshold is even tighter: around 200 milliseconds. That's the window in which humans naturally interject, agree, or respond in fluid conversation. Trying to squeeze a full speech-to-text pass, model inference, and a text-to-speech response through a network round-trip in 200ms is extremely difficult with today's infrastructure.
Switching to a voice-in, visuals-out model buys you meaningful breathing room. You're now targeting that more forgiving 1-second visual response window rather than the brutal 200ms conversational window. That's still fast, but it's achievable today without waiting for novel model architectures.
How Do You Build a Voice Agent That Acts in Real Time?
At Forest Walk, the team built a voice agent that lives inside their calls and takes action on spoken intent — without interrupting the conversation. In one example, during a call between co-founders, one mentioned a bug in the Slack integration. The other confirmed she'd seen it too. One founder simply said, "Let's file that as a Linear issue," and within a second, the agent had done exactly that.
What makes this feel natural rather than gimmicky is that the AI doesn't need to respond with its own voice. It just acts. The spoken intent is captured, interpreted, and executed — silently, on screen — while the human conversation continues uninterrupted. The AI is a participant in the workflow without being a speaker in the conversation.
This pattern — ambient voice intent capture with visual confirmation — is going to become increasingly common as the tooling matures. But getting it to actually work requires solving the latency problem precisely.
How Do You Reduce Latency in Voice AI Apps?
There are three concrete techniques that make the difference between a sluggish prototype and a genuinely responsive experience.
1. Use a Fast Model on a Latency-Prioritized Platform
This sounds obvious, but it's easy to get wrong. When GPT-4o mini launched, there was excitement that a smaller, cheaper model would naturally be faster. In practice, P95 latencies of 5,000 to 10,000 milliseconds were common — nowhere near the sub-second target. Cheaper isn't the same as fast.
Anthropic's Claude Haiku is a much better fit for real-time use cases because of its consistently low P95 latency. The class of model you want is something like Haiku — small enough to respond in a few hundred milliseconds, on an inference platform that actually prioritizes latency as a metric.
For heavier tasks that require deeper reasoning, the architecture should offload that work asynchronously to a larger model. The real-time model handles the fast response and coordinates; the bigger model does the heavy lifting in the background and feeds results back in when ready.
2. Send Inference Eagerly — Don't Wait for Silence
Traditional voice application design waits for the user to finish speaking, detects a second of silence, and then fires off a request. By the time you've waited for that silence, you've already burned through most of your latency budget before the model has even started thinking.
A better approach is to send inference requests every one to two seconds while the user is still speaking. Yes, you might send a request before the person has finished their thought. That's okay. The experience feels dramatically more responsive because the system is processing in parallel with the speaker rather than waiting politely for them to finish. It mirrors how a sharp human collaborator listens — actively, in real time, not waiting for the full monologue before engaging.
How Does Prefix Caching Speed Up LLM Inference?
Prefix caching is one of the most impactful — and underused — techniques for making LLM-powered applications both faster and cheaper. The idea is straightforward: if the beginning of the context window you send to the model is identical across requests, the inference platform can cache that prefix and skip reprocessing it. Depending on the platform and conditions, this can reduce inference costs by up to 90% and meaningfully cut latency.
For real-time voice AI, this means structuring your context window so that roughly the first 90% is stable and consistent from request to request — system prompt, user profile, tool definitions, session state — and only the final 10% changes with each new turn. Combined with minimizing the number of output tokens in each response, this gives you the fast, affordable inference turns that make the experience feel alive rather than laggy.
This architecture applies whether you're building a long-running agent that persists across sessions or a frequently triggered agent that fires on short intervals. The principle is the same: stabilize the prefix, minimize the delta, and keep output tokens lean.
Which AI Models Are Fast Enough for Real-Time Use?
Based on real-world building, the bar for a real-time voice AI model is not just raw capability — it's P95 latency on an inference platform that treats speed as a first-class concern. Claude Haiku currently stands out for this use case. Smaller open-source models running on appropriately optimized infrastructure can also qualify.
The general framework: use the fastest capable model for real-time response, and architect asynchronous handoffs to larger models like Claude Sonnet or GPT-4o for tasks that genuinely require deeper reasoning. The real-time model orchestrates; the heavy model computes. Together, they deliver both speed and intelligence — without asking either model to do what it's not optimized for.
The Bottom Line on Building Voice AI That Feels Good
The gap between voice AI that feels magical and voice AI that feels broken is almost entirely a latency problem. The models are capable. The speech-to-text pipelines are solid. What separates a great experience from a frustrating one is whether you've engineered the full stack — model selection, inference cadence, and caching strategy — to deliver a response within the one-second visual attention window.
Voice in, visuals out is a genuinely compelling interaction model. It works with how humans naturally communicate and how we naturally process information. The architecture to support it exists today. The builders who figure out how to make it feel effortless are going to create some of the most compelling AI products of the next few years.
