Don't clear a backlog at full speed
Most of the APIs we consume enforce a budget we do not control: Shopify's cost-based Admin API limits, a vision model we call for invoice lines, PrintNode when a shop catches up after a quiet hour. When a webhook gap or a failed worker leaves a backlog, the instinct is to drain it at full speed. That is how the live path — the next order, the next parse the user is waiting on — starts getting 429s from work that was supposed to be recovery.
A gap makes you want to sprint
The failure mode is familiar. Webhooks were quiet for forty minutes. A worker crashed overnight. A deploystuck a queue. Now Mongo has holes, the shop is missing tickets, or yesterday's invoices never reached the parser. Someone opens a reconciliation job and points it at the missing range.
That job is usually written for correctness first: page the platform, upsert every record, kick the downstream work. Correctness is right. Throughput is where it goes wrong. The same credentials that serve the live webhook path and the staff tools are now burning request budget on historical catch-up, and the platform cannot tell that your intent is virtuous.
On Al's Flowers the cost shows up as Shopify 429s during a busy window, or PrintNode jobs stamped out so fast the workroom printer becomes a second problem. On BuilderHelp-shaped OCR, a weekend of unparsed supplier invoices turns into a fan-out of vision calls that crowds out the interactive parse a contractor is staring at.
One quota. Several callers who do not know each other.
Rate limits are almost never scoped the way your architecture diagram is. Live webhook handlers, scheduled reconciliation, admin 'sync now' buttons, and batch AI parse workers often share one app credential and one bucket. Each piece has its own retry loop. None of them can see the others' in-flight work.
That is why a polite exponential backoff on every client still fails. Twenty workers each backing off independently still re-arrive together. The platform emptied the bucket once; your recovery rebuilt a synchronized herd. Retry-After tells you when you may try again. It does not decide who among your own processes gets the next slot.
Treat the budget as a shared resource with an owner in your code — a limiter, a worker pool, a queue with a concurrency cap — not as a per-request afterthought in each caller. If catch-up and live traffic cannot negotiate, catch-up will win on volume and live traffic will lose on latency.
Admit fast. Process at a budget.
Inbound webhooks should still ack quickly; that part does not change. What changes is everything after the durable write. Persist the fact that work exists, then drain it through a worker that knows the upstream cost of each step — Shopify REST call cost, GraphQL query cost, tokens on a model call, pages through a printer API.
Cap concurrency on purpose. One or two in-flight reconciliation pages beat ten. Read RateLimit-Remaining and Retry-After when the platform sends them, and sleep with jitter rather than on a fixed metronome. When remaining headroom is low, stop starting new catch-up work even if the backlog is ugly. An honest delay beats a self-inflicted lockout.
Bound the backlog itself. A queue with no max length turns an outage into an unbounded replay against a partner. Prefer a ceiling, a dead-letter path, and a human-visible depth metric over a silent list that grows until Monday.
Live work outranks catch-up
Priority is the part most backlog scripts skip. A webhook that just arrived for today's order should not wait behind a backfill of last Tuesday. A user-triggered invoice parse should not sit behind a bulk re-process of every fuzzy PDF from the last month.
We split lanes when the volume justifies it: a high-priority worker for live ingest and interactive jobs, a low-priority worker for reconciliation and re-parses, both drawing from the same token bucket so neither can ignore the budget. When the bucket is empty, the low lane pauses first.
If you only have one worker today, encode the rule as selection order: always pop live before catch-up. The implementation can be a sort key. The product decision is that recovery is not allowed to starve the path a customer or a shop is waiting on right now.
What this is not
This is not a substitute for idempotent webhook handling or client-minted idempotency keys on mutations. Those keep duplicates from becoming double tickets and double charges. Backpressure keeps recovery from becoming a denial of service against yourself. You still want all three.
It is also not an argument for skipping reconciliation. Gaps happen. The fix is a paced pull with a clear stop condition — not a hope that webhooks were complete. Pace and priority are how reconciliation stays safe on a Monday morning.
If you change one thing on an existing integration, put catch-up behind a concurrency limit that shares headroom with live callers, honor Retry-After with jitter, and make live jobs win the queue. Full-speed backfill feels like diligence. Upstream, it looks like an attack you wrote yourself.