Sub-50ms text-to-speech and why latency budgets decide voice UX

TL;DR: The number that matters for voice agents is not how fast your TTS model runs, it is the total time from the user finishing a word to the first sound coming back, and sub-50ms on one component only helps if the rest of the pipeline is already tight.

The primary source here is a Hacker News post titled “How we made a text-to-speech model respond in sub-50 ms.” I want to be upfront: the material I have is the headline and the framing, not the full engineering breakdown. So I am not going to pretend to relay benchmarks that were not put in front of me. What I can do is talk about why sub-50ms is the right thing to chase, where that number usually goes to die in a real system, and how a builder should reason about the whole latency budget instead of one flashy figure.

Why does 50ms matter for a voice model?

Human conversation has a rhythm. In natural speech between two people, the gap between one person stopping and the other starting is roughly 200ms on average, and pauses longer than about half a second start to feel like hesitation or a bad connection. That is the bar a voice agent is quietly being judged against. Not “is it accurate,” but “does it feel like talking to something alive.”

Fifty milliseconds for time-to-first-audio is aggressive in a good way, because it leaves headroom. If your TTS can start producing sound 50ms after it receives text, you have budget left over for everything else that has to happen first: hearing the user, deciding they are done talking, running the language model, and only then generating speech. The moment one stage blows its budget, the whole thing feels laggy no matter how fast the others are.

This is the trap in celebrating a single sub-50ms component. Latency is additive and, worse, some of it is serial. You cannot start speaking the answer until the language model has produced at least the first token, and the language model cannot start until you have decided the user finished their sentence. A blazing TTS bolted onto a slow turn-detector is like a fast car stuck behind a slow one on a single-lane road.

What actually eats the latency budget in a voice pipeline?

Walk the round-trip. A user speaks. Their audio streams to a server. Speech-to-text transcribes it, usually incrementally. Then something has to decide the turn is over, which is its own quiet villain: end-of-turn detection often adds a few hundred milliseconds of waiting just to be sure the person is not mid-thought. Then the language model reads the transcript and generates a response, and time-to-first-token there is frequently the single largest chunk. Only then does TTS get the text and start producing audio, which streams back to the user and has to play out through their device.

Add those up and a “fast” system can still land north of a full second before the user hears anything. The TTS being 50ms versus 150ms is real, but it is a small slice of that pie. Which is why I read a sub-50ms TTS claim as necessary but not sufficient. It tells me the team took latency seriously on their piece. It does not tell me the product feels responsive.

There is also a measurement honesty problem. Sub-50ms measured how? Time to first audio sample, or time to a fully rendered utterance? On what hardware, on-device or in a datacenter, and does that number include the network hop to the user? Cold start or warm? A model that hits 50ms warm on a rented H100 in the same rack as the caller is a very different thing from one that does it on a phone over cellular. None of that is a knock on the claim. It is the set of questions you have to ask before the number means anything for your use case.

How should a builder think about the whole budget?

Set a target for the felt experience first, then divide it. If you want responses that feel conversational, aim for time-to-first-audio under roughly 500 to 800ms end to end. Then back out how much each stage gets. Streaming STT that emits partial transcripts. Turn detection that is tuned to your domain instead of a generic silence timeout. A language model where you have measured time-to-first-token, not just tokens-per-second, because for the felt latency the first token is what unblocks speech. And TTS that streams audio chunk by chunk rather than waiting to synthesize the whole sentence.

That last point is where a sub-50ms TTS earns its keep. If your speech model can start emitting audio almost immediately after the first few tokens of the LLM response arrive, you can overlap generation and playback. The user hears the beginning of the answer while the rest is still being written. Done well, that overlap hides more latency than any single optimization, because the human perception of speed is anchored on when sound starts, not when it finishes.

The other move most people skip: measure against real users on real networks, not localhost. A pipeline that is snappy on your machine can fall apart once you add the round-trip to a phone in a coffee shop. Latency you cannot see in dev is still latency the user feels.

I would love to have the full engineering detail from that Hacker News writeup, and if it holds up, sub-50ms TTS is a genuinely useful building block. Just do not let one impressive component number stand in for the thing your users actually experience.

If you are building a voice agent, do this: instrument every stage and log the timestamp at each handoff, from audio-in to STT partial to turn-decision to LLM first-token to TTS first-audio to playback. Look at the total, find the fattest segment, and fix that before anything else. A sub-50ms TTS is a great card to hold, but the catch most readers miss is that shaving 100ms off an already-fast stage is invisible while a lazy turn-detector or a slow LLM first-token quietly costs you half a second every single turn. Optimize the budget, not the headline.