Redesigning an Integration After Hitting a Vendor's Undocumented Rate Limit

A nightly sync job pulling customer records from a CRM vendor's REST API worked perfectly in staging, where we tested against a sandbox account with a few hundred records. In production, against an account with eighty thousand records, roughly 3% of records silently failed to sync every night, with no error in our logs, no failed request, nothing. Just gaps that only showed up when someone cross-referenced the record counts weeks later.

What was actually happening

The vendor's documented rate limit was 100 requests per minute, and our client correctly throttled to stay under that. What wasn't documented anywhere: the API also enforced an undocumented secondary limit on a rolling 24-hour window for a specific endpoint we were hammering for incremental updates, and once you hit it, instead of returning a 429 error, it returned a 200 response with a silently truncated result set. No error. No header indicating truncation. Just fewer records than existed, presented as if it were the complete, successful response.

A rate limit that fails with an error is an inconvenience. A rate limit that fails by silently truncating a 200 response is a data integrity incident waiting to be discovered by someone else.

How we actually found it

Not from the vendor's support team, who initially insisted there was no secondary limit. We found it by instrumenting our own request/response counts and comparing the response record count against the account's known total record count on every sync — a check we added only after noticing the discrepancy manually. Once we had that metric, the pattern was obvious: truncation correlated exactly with request volume in the preceding 24 hours, cliff-edged at a threshold the vendor eventually, reluctantly confirmed existed but had never published.

• Added a record-count reconciliation check as a permanent part of the sync job — compare returned count against expected total, alert on any gap, don't just trust a 200.
• Redesigned the sync to request smaller incremental windows spread more evenly across 24 hours instead of a single nightly batch, staying well under the undocumented ceiling.
• Added a cursor-based checkpoint so a partial sync could resume from where it left off instead of needing a full re-pull.
• Stopped trusting vendor rate-limit documentation as the ceiling — now budget for an unknown secondary limit by design, with reconciliation checks as the actual safety net, not the docs.

The fix took about three days once we understood the actual failure mode. Finding it took three weeks, because a silent 200 with truncated data doesn't look like a bug — it looks like success. Any integration pulling meaningful volume from a third-party API needs its own count-based sanity check, because the vendor's documentation telling you about their limits isn't the same as the vendor's API actually behaving the way the documentation says.