A hard cutover is not a secret rotation
Idempotent ingest and reconciliation keep a florist's print pipeline honest when Shopify retries or goes quiet. They do nothing when your verifier rejects a valid signature because someone rotated the signing secret and the old one died before the deploy finished. A rotation is an overlap window, not a hard cutover.
Secrets do not stay put
Signing secrets leave the place you put them. A laptop goes home with a teammate who is leaving. A CI log echoes an env dump. A staging URL that still points at production credentials gets bookmarked. GitHub's secret scanner will catch a committed key in minutes — treat that as compromised the moment it lands, not when you finish reading the email.
At Al's Flowers the webhook path is the front door to a physical printer. An invalid-signature spike is not a polite 401 in a log. It is a silent gap in the ticket stream until someone notices the bench is empty. Rotation is a scheduled operation, not a panic button you invent under that pressure.
Planned rotation and emergency revocation are different jobs. Planned rotation keeps both secrets valid long enough for every instance to catch up. Emergency revocation kills the leaked key immediately and accepts a short delivery hit — then you lean on the reconciliation pull you already built for missed webhooks.
The overlap window is the whole trick
The failure mode is a same-minute swap: generate a new secret in the Shopify or Stripe dashboard, paste it into production, redeploy. Every event signed with the old secret that is still in flight — or still being retried — fails verification. Your handler returns 401. The platform retries. Those retries keep failing until the window expires or someone rolls back.
The fix is a dual-secret verifier. Store current and previous (or current and pending) in the secrets manager. On each delivery, try each secret with a constant-time compare — crypto.timingSafeEqual in Node, hmac.Equal elsewhere — and accept the first match. Log which label matched without logging the secret or the raw signature.
Providers that already support rolling secrets make this easier. Stripe keeps the old signing secret valid for a grace period after you roll. Shopify app credentials and custom receivers vary; if the platform only exposes one active secret at a time, you still own the receiver side: deploy multi-secret verification before you click rotate, and keep the old value in your env until delivery metrics prove the new one is carrying traffic.
A rotation runbook that fits a small team
Generate the new secret. Add it alongside the old one in the secrets manager — do not delete the old value yet. Deploy (or hot-reload) a verifier that accepts either. Confirm a test delivery verifies; watch an attribute for matched=current vs matched=legacy.
Promote the new secret in the provider dashboard (or switch the sender to sign with it). Wait until matched=legacy drops to zero for a full retry window — hours for a quiet shop, longer if platforms retry for a day. Only then remove the old secret from the receiver and redeploy the single-secret path.
Fail closed on empty config. A boot that cannot find WEBHOOK_SECRET_CURRENT should refuse to start, not verify against an empty string. During the overlap, missing the pending key is also a deploy bug — treat it like a missing database URL.
Verify the bytes you signed, not the JSON you parsed
HMAC verification runs on the raw request body. Frameworks that parse JSON first and re-serialize it change whitespace and key order; the signature will not match even when the secret is right. Read the raw bytes, verify, then parse. This shows up in every Shopify and Stripe integration guide for a reason.
Timestamp skew belongs in the same gate. Reject deliveries whose signed timestamp is outside a short window so a captured payload cannot be replayed weeks later. That check is independent of which secret matched — both secrets use the same skew policy during overlap.
Invalid signatures should be loud and boring: a stable error, a metric, and no side effects. Do not enqueue print jobs, do not write order rows, do not call the platform back. A flood of bad signatures after a botched rotation is how you notice the runbook failed; a quiet accept of unsigned traffic is how you notice too late.
What this is not
This is not a substitute for idempotent ingest. Rotation keeps verification from dropping valid traffic. Idempotency on the order id still catches the duplicate that arrives twice after a retry. Reconciliation still catches the delivery that never arrived. The three layers answer different lies.
It is also not an argument for keeping two secrets forever. The overlap window exists to finish the cutover. A permanent dual-secret setup doubles the blast radius of a leak. Set a calendar reminder when you start a rotation, and purge the legacy key when the metrics say it is idle.
Cursor-signed pagination tokens and webhook signing secrets are related ideas with different lifetimes. A cursor HMAC can rotate on its own schedule with a verification-failure restart. A webhook secret rotation has to survive in-flight platform retries. Do not collapse them into one env var and one runbook.