Generate the idempotency key when they click, not when you retry
Most of the APIs we ship are small mutation surfaces: create a print job, start a payment, submit a vote, kick off a parse. Clients retry those calls because mobile networks drop, serverless cold-starts time out, and operators mash the button. Without an idempotency key minted when the user acted — and reused on every attempt — a retry is not recovery. It is a second charge, a second ticket, or a second row that looks legitimate until someone notices in the shop.
Timeouts are the failure mode that creates duplicates
A GET that times out is annoying. A POST that times out is ambiguous. The server may have committed the write and lost the response on the way back, or it may have never seen the request. From the client's seat those two outcomes look identical: no body, no status, and a user still waiting.
The instinctive fix is to retry. That is correct for network failure and wrong for side effects unless the server can recognize the second attempt as the same action. Stripe popularized the pattern for payments; the same shape shows up anywhere a mutation matters — order ingest, ticket creation, provisioning, anything that spends money or produces paper.
On Al's Flowers the physical side makes the cost obvious. A duplicate print job is two arrangements or a wasted ticket. On BuilderHelp a duplicate payment attempt is worse. The API has to treat 'same intent, second attempt' as a first-class case, not an edge case you hope the client avoids.
Mint the key at intent, reuse it on every attempt
The key must identify the user's action, not the HTTP attempt. Generate a UUID when they tap Pay, Submit, or Print — before the first request leaves the device — and attach the same value on every retry of that action. Regenerating the key in the retry loop defeats the whole mechanism: each attempt looks brand new, and you get exactly the duplicates you were trying to prevent.
We send it as an Idempotency-Key header on unsafe methods. The server stores the outcome keyed by caller and key: status code, response body, and enough of the request fingerprint to reject conflicts. A later request with the same key and the same payload returns the stored response. The same key with a different body is a client bug — answer with 409 or 422, not a silent re-execution.
Scope the key to the authenticated caller. Otherwise two users can collide on a poorly generated value, or a leaked key can replay someone else's result. For staff tools and mobile clients we already have a session; bind the store to that identity.
Insert first. Do not check-then-act.
The race that bites is two retries in flight at once — a mobile client with aggressive backoff and a user who taps again. Both requests arrive before either has finished writing. A SELECT-then-INSERT in application code lets both checks pass. The uniqueness has to live in the store: try to claim the key with an insert (or SET NX), and let the constraint decide the loser.
What you store matters. Cache successful responses and stable client errors that represent a finished decision. Be careful with 5xx and timeouts of your own: if you persist a transient failure against the key, every retry until TTL expiry replays that failure instead of finishing the work. Our default is to reserve the key while processing, commit the outcome on a completed response, and leave the key claimable again if the handler crashed before commit.
TTL should cover the client's real retry window, including rate-limit backoff. Twenty-four hours is a boring default for payment-shaped mutations. Five minutes is too short if a throttled client comes back an hour later and looks like a new action.
Rate limits and idempotency are one system
Clients that retry without keys create load. Rate limits without Retry-After create guesswork. The two mechanisms work together: throttle how often a caller can start new work, and make it safe to repeat work already started.
When a request carries an idempotency key you have already completed, return the cached response even if the caller is over their request budget. Charging them a 429 for a replay they are required to do pushes them toward minting a new key — which is how you get the duplicate you were rate-limiting to prevent. New intents can wait. Replays of settled intents should be cheap and consistent.
Put RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset on responses where you can, and always put Retry-After on 429. Document which status codes are safe to retry, and that retries must reuse the key. A client library that regenerates UUIDs in its retry middleware is a liability; we treat that as a review blocker the same way we treat an unpinned Shopify API version.
What this is not
Inbound webhooks are a different problem. There the platform already has a stable event or order id, and your job is to unique on that and reconcile silence. We wrote about that for the Al's Flowers print path. Idempotency-Key headers are for the APIs you expose when the client is the one who knows the intent and the network sits between click and commit.
Natural keys help when they exist — an order id, a content hash, a vote in a room. Use them. Keys are for the cases where the client creates something that does not have a stable id until after the server responds, which is most POST create endpoints.
During scoping we ask: which mutations are unsafe to double, who retries them, and where the key is minted. If the answer is 'the HTTP client library, on each attempt,' the design is already wrong. Fix that before the first timeout in production teaches the same lesson with a duplicate ticket.
Questions
- When should a client generate the Idempotency-Key?
- When the user intent happens — tap Pay, Submit, Print — before the first HTTP attempt. Reuse that same key on every retry of that action. A new UUID per attempt disables deduplication.
- How do you avoid races between concurrent retries?
- Claim the key with an insert or SET NX and let the unique constraint decide. Check-then-act in application code loses under concurrent redelivery; both checks pass before either write lands.
- Should a replay hit the rate limiter?
- No. If the key already has a settled response, return it even when the caller is over budget. Rate-limit new intents; do not punish required replays into minting a new key.
Sources
- Stripe — Idempotent requests
- IETF draft — RateLimit header fields
- Shopify webhook best practices — Related inbound pattern; use platform event ids there, not client-minted keys.