Lessons From Rate-Limiting Our Own Internal API

An internal reporting API started getting hammered by a misconfigured retry loop in a client's internal automation script — not malicious, just a bug that retried every failed request immediately with no backoff, turning one legitimate slow request into about forty requests a second from a single source. We built rate limiting to stop this from ever taking down the API for everyone else. The first version we shipped was wrong in an instructive way.

The first version solved the wrong problem

We implemented a flat global rate limit — 1,000 requests per minute across all clients — reasoning from total traffic capacity. It stopped the retry-loop incident from recurring, but it also meant that one high-volume legitimate client could exhaust the entire budget and get other, smaller clients throttled for requests that had nothing to do with the actual problem. We'd designed around 'how much traffic can the system handle' instead of 'what failure mode are we actually trying to prevent,' and those turned out to be different questions with different correct answers.

A rate limit designed around total capacity protects the server. A rate limit designed around the actual failure mode protects everyone else's legitimate traffic from one bad actor.

What we rebuilt it as

Per-client rate limits, scoped to an API key, with a burst allowance and a sliding window — so a single misbehaving client gets throttled in isolation, and everyone else is unaffected. We also added a specific, tighter limit on repeated identical requests within a short window, because the original incident wasn't really about volume, it was about a retry loop hammering the exact same failed request over and over with no backoff.

Per-API-key limits:
  sustained: 100 req/min (sliding window)
  burst: 20 req/10s
  identical-request dedup: max 3 retries/min on same payload hash

Global fallback ceiling exists but is sized well above any single
legitimate client's expected peak, purely as a last-resort circuit breaker.

• Scope limits per client, not globally, unless you specifically want one client's misbehavior to degrade service for everyone else.
• Add a specific limit on repeated identical requests, since a misbehaving retry loop looks different from genuine high-volume usage and deserves a different, tighter rule.
• Return a clear 429 with a Retry-After header and actual guidance — a silent drop or a generic error just pushes the client to retry harder, which is exactly the failure mode you're trying to prevent.
• Log and alert on clients hitting their limit repeatedly — that's a signal of a bug on their end worth proactively flagging, not just a problem to throttle away silently.

The retry-loop client, once we could identify them specifically from the per-key logs, turned out to have a one-line bug in their backoff logic that they fixed within a day once we told them exactly what was happening. Rate limiting isn't really about defending a fixed amount of server capacity — it's about shaping what happens when something goes wrong on the other end, and a flat global number doesn't know the difference between popularity and a bug.