A Webhook Retry Storm Took Down a Client's Tracking Pipeline

A downstream API we depended on had a two-minute blip — a deploy on their end caused a brief spike in 503 responses. Two minutes should have been a non-event. Instead it turned into a four-hour outage of a client's tracking pipeline, because the upstream vendor sending us webhooks had aggressive retry logic with no backoff, and our own receiving endpoint had no rate limiting or queuing in front of it.

How a two-minute blip became four hours

Every failed webhook delivery during that two-minute window got retried by the sending vendor, repeatedly, without exponential backoff. By the time our downstream API recovered, we had a backlog of retried webhooks arriving at several times normal volume, which our own processing layer wasn't provisioned to absorb in a burst. That caused our endpoint to start timing out under the load, which triggered further retries from the vendor, which fed the same loop. The original two-minute downstream issue was long resolved; the retry storm it triggered kept feeding itself for hours after.

A retry storm doesn't need a long outage to start. It needs a short one, hit with enough retry pressure to outlast the original problem on its own momentum.

What actually fixed it and what we changed after

• Immediate fix: manually throttled the incoming webhook endpoint and let the backlog drain gradually instead of trying to process it all at once.
• Added a message queue in front of the webhook processing layer, so bursts get buffered and processed at a steady rate instead of hitting the processing logic directly.
• Implemented our own circuit breaker on the downstream API call, so a failure there fails fast and queues for retry on our terms instead of blocking and timing out under load.
• Documented the vendor's retry behavior explicitly, since their docs didn't mention the lack of backoff, and we only learned it by reading the actual retry timestamps in our logs.

The root cause wasn't the vendor's two-minute blip, and it wasn't entirely their lack of backoff either — it was that our own system had no buffer between an unpredictable inbound load pattern and a processing layer that assumed steady-state traffic. That gap is exactly where a short, unremarkable incident somewhere else turns into your own multi-hour outage.