An AI Agent Latency Budget We Didn't Have Until Users Complained

We built an agent that chained three tool calls to answer a user question — a search, a lookup, and a synthesis step. Every individual step was correct. The combined latency, on a bad day, hit nine seconds. Nobody had set a latency budget, because correctness was the thing we'd been testing for, and nine seconds of correct felt fine in a demo.

What users actually did

Session recordings showed a pattern we hadn't anticipated: a meaningful fraction of users re-submitted the same query after four or five seconds of silence, assuming it had failed. That doubled the actual load on the system and, worse, meant the UI was quietly training users to distrust a feature that was working correctly the entire time — just too slowly to feel like it was working.

A correct answer that arrives after the user has already assumed failure is, from the user's perspective, indistinguishable from a wrong one.

What we did about it

• Set an explicit p95 latency budget per agent step — a number we should have had before launch, not after complaints.
• Added incremental UI feedback showing which tool was currently running, so silence never exceeded about 1.5 seconds between visible updates.
• Parallelized the search and lookup calls, which had no actual dependency on each other despite being written sequentially out of habit.
• Added a hard timeout with a graceful partial-answer fallback instead of letting a slow tool call hold the whole response hostage.

The parallelization alone cut median latency by about 35%, and the incremental feedback did more for perceived speed than any of the backend changes. We now treat latency budget as a launch requirement for any agent feature, defined before the first tool call is wired up, not discovered after the first wave of confused users.