Fixing Latency in a Text-to-Speech Pipeline That Felt Unusably Slow

User feedback was consistent and vague: the voice assistant 'felt slow,' even annoyingly so. The TTS model's own benchmark numbers said otherwise — sub-200ms generation time for a typical response. Something between the model and the user's ear was eating all the perceived speed, and nobody had actually profiled the full pipeline end to end before assuming the model was the bottleneck.

Where the time actually went

Once we instrumented the whole request path, the picture looked nothing like the model benchmark suggested. The LLM generating the response text took 1.8 seconds before TTS could even start. The pipeline was waiting for the full text response to complete before sending anything to the TTS engine, instead of streaming partial text as it generated. And the audio playback layer buffered the entire clip before starting playback instead of streaming audio chunks as they were produced. The TTS model's 200ms was real and almost irrelevant — total time to first audible word was averaging 2.6 seconds.

The model everyone blamed was the fastest part of the pipeline. The slowness was three sequential waits nobody had profiled.

What actually closed the gap

• Streamed LLM tokens into TTS generation as they arrived instead of waiting for the full response text.
• Started audio playback on the first generated chunk instead of buffering the complete clip.
• Pre-warmed the TTS model to avoid cold-start delay on the first request of a session.
• Measured and reported time-to-first-audible-word as the primary latency metric instead of raw model inference time, since that's what users actually perceive.

Time to first audible word dropped from 2.6 seconds to about 400 milliseconds, almost entirely from eliminating sequential waits rather than making any individual component faster. The lesson generalizes well past voice: whatever latency metric a vendor benchmarks is almost never the latency metric your user experiences, and the gap between them usually lives in your own integration code.