A Streaming UI Made a Slow LLM Response Feel Fast, UX Over Engineering

After exhausting the backend optimizations — batching, caching, parallel tool calls — a particular LLM response was still taking around seven to nine seconds for longer outputs. We'd hit the floor of what was realistically achievable without swapping to a smaller, worse model. The fix that actually worked wasn't an engineering fix at all, it was a UX one.

What we changed

We switched from returning the full response at once to streaming tokens as they generated, rendering text progressively instead of showing a spinner for nine seconds and then dumping the whole answer. The total time to full completion didn't change by a single millisecond.

We didn't make the response faster. We made the first token arrive in 400 milliseconds instead of the full answer arriving in nine seconds, and that was the only number users actually felt.

What changed in the data

• Perceived-wait complaints dropped by roughly 70% in the following month's support tickets, despite zero change to actual generation time.
• Session abandonment during generation dropped noticeably — watching words appear kept people engaged in a way a static spinner never did.
• Re-submission of the same query (users assuming a stall) nearly disappeared, since visible progress removed the ambiguity between 'working' and 'broken.'

This isn't a universal fix — streaming doesn't help when the useful part of the answer is the conclusion, not the build-up, and some outputs genuinely read worse word-by-word than as a complete block. But for most conversational and explanatory responses, the gap between actual latency and perceived latency is wide enough that closing it with UI work is cheaper and faster to ship than another round of backend optimization.