Every fix ships with the test that would have caught it
When something breaks in production, there are two things to salvage: the fix, and the input that caused it. Most teams keep the first and discard the second. We've found that keeping both — and naming the test after the real document that exposed the bug — changes how a codebase ages.
The input is the valuable part
We maintain a tool that reads vendor purchase orders. Every vendor formats their PDFs differently, and the parsing layer's field naming is not deterministic — the real part code shows up under a different key depending on the document.
So the failure modes multiply. The code leads the description instead of sitting in its own column. The code is buried mid-description. OCR turns zeros into the letter O. The part is legitimately all-numeric and looks like a stock number. Quantity and price columns are swapped.
Each of those is a separate recovery pass. And each one ships with a test built from the actual purchase order that exposed it, with the real document identifier in the test name. When a regression fires, it doesn't say "parser test 14 failed" — it points straight at the case it broke.
| What reality sent | The recovery | The test fixture |
|---|---|---|
| Code leads the description | extractLeadingPartCode | The Winnelson PO that exposed it |
| Real part is all-numeric | filterKnownCodes | The cartridge order, 9745299 |
| OCR turned 0 into O | Typo variants, 6+ char tokens only | The scanned order it came from |
The document identifier goes in the test name, so a regression points at the case it broke rather than at 'parser test 14'.
A human correction is ground truth
The most useful telemetry we get isn't an error log. It's a correction.
When a purchase order parses wrong, the user sends a diagnostics bundle: the raw parser response, the OCR text, and the stored line items including any edits a person made. That last part is the gold. A human correcting a line is an authoritative statement about what the document actually said — better than any heuristic we could write, and generated for free by someone doing their job.
Designing the system to capture corrections, rather than just overwriting them, turned support requests into a labeled dataset. The design consequence is small but worth naming: corrections have to be stored as corrections, not written over the parsed value. If the edit simply replaces the original, the record of what the machine got wrong disappears at exactly the moment it becomes useful. Keeping both costs a column and buys the entire feedback loop.
The invariant matters more than the fix
Aggressive error recovery is dangerous in a way that's easy to miss. A heuristic clever enough to pull a part number out of a description is also clever enough to turn an ordinary English word into a bogus part number — and that error looks like valid data all the way downstream.
So every recovery pass obeys two rules without exception. It only touches lines whose existing code is already useless. And it must confirm a candidate exists in the manufacturer's database before adopting it. Fuzzy matching tries exact matches first, and only considers typo variants for tokens long enough that a coincidence is implausible.
That constraint has held through every new vendor format. It's the difference between a tool that saves time and one that quietly manufactures wrong orders — and it's worth more than any individual fix we've shipped.
Saying no to the configurable version
Users regularly ask for a per-vendor template: a place to tell the system where this particular vendor puts the part number.
It's a reasonable request and we decline it. The honest answer is that the fixes are general and need no configuration — a new vendor's PDF usually falls into a shape we've already handled. And a template fundamentally cannot repair a code that OCR mangled before any mapping logic sees it. Adding the feature would create a maintenance surface, a support burden, and a false sense that the problem is solved.
Telling a client the feature they asked for won't work is uncomfortable. Building it anyway and watching it not work is worse.
Where this discipline doesn't pay
We don't do this everywhere, and claiming otherwise would be dishonest. A fixture built from a real input is worth it when the input space is adversarial and unbounded — vendor documents, user-generated content, anything from OCR or a third-party API you don't control. Reality keeps inventing new shapes there, and each one you capture is a shape that can't come back.
It's overhead when the input space is small and you already understand it. A form with four fields does not need a fixture per submission. A pure function with three branches needs three tests, written from the branches rather than from a bug report. Building elaborate fixture infrastructure around code like that is a way of feeling rigorous without being safer.
The signal we use is whether the bug surprised us. If we could have predicted the input from reading the code, the fix is just a fix. If reality produced something we didn't know was possible, that input is information we don't otherwise have — and information that expensive is worth keeping.
What this costs and what it buys
It's slower per fix. Reproducing a real document as a fixture takes longer than patching the symptom, and one change becomes one pull request rather than a batch of five. There's also a real temptation to batch: five small fixes in one branch is faster to write and much harder to bisect when one of them turns out to be wrong.
What it buys is a codebase where the accumulated knowledge lives in the tests rather than in whoever happened to fix it. New failure shapes get compared against existing ones instead of rediscovered. And when someone eventually inherits the project, the test suite reads as a documented history of everything reality threw at it.
That's the version of a codebase we want to hand over — one where the hard-won knowledge is written down, in the only place that fails loudly when it's forgotten.