Webhook ingestion that assumes the network lied
Webhook handlers should treat every delivery as a rumor: it may arrive twice, late, or never. At Al's Flowers, Shopify orders become PDFs and PrintNode jobs for a physical printer — so a duplicate is two arrangements, and silence is a missing Valentine's order. Idempotency on the order ID plus reconciliation against Shopify are what make that pipeline safe.
A printer in the back of a flower shop
The requirement is mundane and unforgiving. An order comes in online, and a ticket needs to come out of a printer where the staff are actually standing, with the right address, the right pickup or delivery designation, and the right arrangement details. Staff work from paper. If the paper is wrong or missing, the order does not happen correctly.
What makes this different from a normal integration is that the output is physical and irreversible in the way physical things are. A duplicate database row is a cleanup task. A duplicate ticket is two people making the same arrangement, or one arrangement thrown away. A missing ticket is a customer who does not get flowers on the day they cared about.
So the design question is not whether to handle delivery failures. It is what specifically happens when the webhook arrives twice, arrives late, or does not arrive.
At-least-once, which sometimes means twice
Webhook delivery from any platform is at-least-once. Shopify will retry when it does not get a timely success response, and 'timely' is measured from its side, so a handler that does real work before acknowledging can be slow enough to trigger a retry while still succeeding. You then have two deliveries of the same order, both legitimate from the sender's perspective.
The first structural fix is to make acknowledgment cheap. The handler validates the request, persists the raw event, and returns success. Rendering a PDF and talking to a print service happen after, driven by the stored record rather than inline in the request. This removes the class of duplicates caused by your own latency, which is the easiest class to cause and the easiest to eliminate.
It also separates two failure domains that should never have been coupled. If the print service is unreachable, that should not make Shopify think the order was not received.
- 01Webhook arrivesPossibly twice, possibly out of order
- 02Idempotency keySame event id lands once
- 03Store raw firstBefore any interpretation
- 04ProcessFailures retry without re-ingesting
- 05Trailing scanAsk the source what we missed
Delivery guarantees are at-least-once at best. The reconciliation pass is what turns that into exactly-once in practice.
Idempotency on the order, not the request
The key that makes duplicates safe is the Shopify order identifier, not a request id or a timestamp. Two deliveries of the same order carry the same order id, and that is the fact you want to be unique on. A unique index on that identifier means the second insert loses, deterministically, in the database rather than in application logic that races with itself.
This matters more than it sounds because the natural instinct is to check-then-write — look for an existing record, and insert if absent. Under concurrent redelivery, both checks pass before either write lands, and you get two records and two tickets. The uniqueness has to be enforced by the store, and the duplicate has to be a caught error rather than a branch you evaluated earlier.
The state of the record carries the rest. An order that has been received, an order whose ticket has been rendered, and an order whose print job was accepted are distinct states, so a redelivery arriving after printing can be recognized as already handled rather than re-executed.
The step that fails after you said yes
Printing is the interesting failure because it happens at the end, after everything upstream has reported success. The print service accepts a job; the printer is out of paper, or offline, or the job sits in a queue nobody is watching. From the application's perspective the call succeeded.
So acceptance by the print service is recorded as its own state, distinct from the ticket having been rendered, and distinct from any assumption that paper exists. Retries operate on that state — a job that was never accepted can be retried safely, because no paper was produced. A job that was accepted must not be blindly retried, because a retry there is the duplicate-ticket failure.
That asymmetry is the whole design. Retry aggressively before the side effect, never automatically after it. Anything past the point of physical output goes to a person, because the system cannot see the paper.
Reconciliation beats cleverness
The last piece is the one that catches everything the retry logic cannot: periodically compare the orders that exist in Shopify against the orders the service has records for, and surface the gap. It is unglamorous and it subsumes an entire category of bugs, including the ones you did not anticipate. In practice that means a trailing-window scan: ask the source for everything in the last few hours, diff it against what we stored, and emit the missing ids as a replayable list rather than trying to repair state in place.
This matters because the worst webhook failure is silence. A webhook that never arrives generates no error, no log line, and no alert — the absence of a message is indistinguishable from no order having been placed. No amount of handler robustness detects that, because the handler never ran. Only a comparison against the source of truth does.
If you build one thing beyond a correct handler, build the reconciliation. Handlers get the cases you thought of; reconciliation gets the rest, and in an integration where the output is a piece of paper someone is waiting on, the rest is where the actual customer damage lives.
Questions
- Why acknowledge a webhook before doing the real work?
- Platforms retry when they do not see a timely success. Doing PDF render and print inline makes your own latency look like failure and creates legitimate duplicate deliveries. Persist the raw event, return success cheaply, then process from the stored record.
- What should idempotency key on for Shopify orders?
- The Shopify order identifier, enforced with a unique index in the store — not a request id or a check-then-write in application code. Concurrent redeliveries both pass a soft check; only the database can make the second insert lose deterministically.
- Why is reconciliation required even with a solid handler?
- The worst webhook failure is silence: no error, no log, no handler run. Periodically diff recent Shopify orders against local records and replay gaps. No amount of handler robustness detects a message that never arrived.
Sources
- Shopify webhook best practices
- Stripe — webhook idempotency guidance — Same at-least-once delivery model applies across platforms.
- PrintNode API docs
- MongoDB unique indexes