Idempotency Keys Saved Us From a Double-Billing Disaster

A brief network blip between a client's checkout service and their payment processor caused a batch of payment confirmation webhooks to time out on our side before we'd finished processing them, even though the underlying charge had actually succeeded. The payment processor's retry logic, reasonably, resent those webhooks a few minutes later. Without an idempotency check, every one of those retried webhooks would have triggered our downstream order-fulfillment and billing-reconciliation logic a second time, against charges that had already gone through.

We'd built idempotency keys into this pipeline early, mostly out of general good practice rather than any specific fear of this exact scenario — each payment confirmation carried a processor-issued transaction ID, and our handler checked that ID against a table of already-processed transactions before doing anything else. The retried webhooks hit that check, got recognized as already-processed, and were discarded cleanly with no duplicate fulfillment action and no duplicate billing event sent downstream.

An idempotency key doesn't prevent the retry. It prevents the retry from mattering, which is the only thing that actually matters when a payment is involved.

Why this matters more in martech pipelines than people assume

• Any webhook-based integration — payment processors, CRM updates, CAPI events — will eventually be retried by something, whether due to timeouts, deploys, or network blips.
• The retry itself is usually correct vendor behavior, not a bug — treating it as something to prevent rather than something to handle idempotently is the wrong framing.
• An idempotency key only works if it's checked before any side effect happens, not logged afterward as an audit trail — ordering matters.
• The check needs a durable store, not an in-memory cache, or a server restart between the first attempt and the retry reintroduces the exact bug you built the key to prevent.

What stuck with me is how unglamorous the save was. No alert fired, no incident channel lit up, because the system did exactly what it was built to do. The only reason I know about it at all is that I happened to be reviewing logs for an unrelated reason and noticed the discarded duplicate webhooks sitting there, quietly doing their job.