A customer correction is the most valuable telemetry you have
For document pipelines, a bug report without the document state is nearly useless — nothing threw; the output was just wrong. The PO tool's diagnostics blob carries raw parser response, OCR text, and stored line items including human corrections. Those corrections are labeled ground truth from someone reading the page, and they beat internal guesses about what the vendor meant.
The bug report problem
For most software, a bug report plus a stack trace gets you most of the way. Document pipelines do not work like that, because there is no exception. The parser returned a perfectly valid structure. The matching ran. Nothing threw. The output was simply not what the document said, and the only party who knows what the document said is the person looking at it.
You also cannot reproduce it from a description. 'The part number was wrong on the third line' does not tell you whether the parser put the code in an unexpected field, whether OCR damaged a character, or whether the code was embedded in the description. Those have completely different fixes and identical symptoms.
Asking the user to email the PDF gets you partway. It does not tell you what the parser returned for that PDF at the time, and re-running it later can produce different field naming, so you may not be debugging the same event.
What the blob contains
So the blob carries three things. The raw parser response, exactly as returned, before any normalization — this is what our recovery passes actually received. The OCR text, which distinguishes 'the characters were wrong on the page' from 'the characters were right and we mapped them badly.' And the stored line items, including the human corrections made in the review modal.
Each layer answers a different question. Comparing OCR text to the document tells you whether the damage predates your code. Comparing the raw parser response to the OCR text tells you whether the model dropped or misfiled something that was legibly present. Comparing the final line items to the corrections tells you what should have happened.
Keeping the raw response is the piece people skip, and it is the one that matters most when field naming is non-deterministic. Without it you are guessing at what your own normalizer was handed.
- 01It parses wrongA line comes out mangled
- 02A person corrects itGround truth, generated for free
- 03Diagnostics blobRaw response, OCR text, the correction
- 04Root causeCompared against known failure shapes
- 05One change, one testFixture built from the real document
The correction is the valuable part. Overwriting it instead of storing it throws away the only authoritative signal you get.
Why the correction is ground truth
Here is the part worth internalizing: when a user fixes a line in the review modal, they are not filing feedback. They are reading the document and telling you what it says. That is a labeled example produced by a domain expert with the source material in front of them, which is a description of the highest-quality training signal available anywhere in the system.
It beats our own judgment, too. When we look at a scan and a raw parser response, we are inferring what the vendor meant. The person at the rep firm knows the vendor, the catalog, and often the specific job. Their correction settles questions we would otherwise argue about internally.
This changes how you treat the review modal in the data model. Corrections are stored as corrections, distinguishable from the parsed values they replaced, because the delta is the signal. Overwriting the parsed value in place would have destroyed the most valuable byte in the record.
Root-cause, one change, ship
The loop from there is deliberately narrow. Read the blob, determine which layer failed, reduce it to the smallest input that reproduces, write the test with the real document identifier in the name, add or amend exactly one recovery pass, ship it.
One change per fix, even when the blob reveals three problems. Three fixes in one release means that when something regresses next month you have three candidate causes and no way to separate them. It also means the customer gets a specific claim — this category of document now parses — rather than a vague assurance that parsing improved.
The discipline pays off over time in a way that is hard to see early. After enough cycles, the passes form a documented catalog of every way vendor documents have actually broken, each one tied to a real PO. That catalog is the asset. The parser is replaceable; the catalog of known failure modes is not.
The template question
Users often propose a different solution: give each vendor a template. Map this vendor's columns once, apply it to every future PO from them, done. It is an intuitive ask and we have said no to it consistently.
Two reasons. The fixes we ship are general — the pass that recovers a code from a leading position in the description works on any document where the code leads the description, regardless of vendor, including vendors we have never seen. A template would re-solve that per vendor and require configuration for each new one, which turns onboarding a vendor into a support ticket. And templates do not address the failures that actually hurt. No column mapping repairs a code where OCR replaced a zero with a letter, because the layout was never the problem.
The honest version of that answer is worth giving directly: the thing you are asking for is more configuration and less coverage. What you actually want is for the next PO from a vendor you have never dealt with to work without anyone setting it up, and that only comes from general fixes.
Questions
- What should a document-pipeline diagnostics blob include?
- The raw parser response before normalization, the OCR text, and the stored line items with corrections kept distinguishable from parsed values. Each layer answers whether damage predates your code, whether the model misfiled legible text, or what the correct line should have been.
- Why are user corrections better telemetry than engineer review of the scan?
- The operator knows the vendor, catalog, and often the job. Their fix is a domain expert labeling the source document. Preserve the delta; overwriting parsed values in place destroys the signal.
- Why not solve bad parses with per-vendor templates?
- General recovery passes cover unseen vendors; templates re-solve the same layout per customer and turn onboarding into support. Templates also do not fix OCR character damage, which was never a column-mapping problem.
Sources
- PO processing tool case study
- Tesseract OCR documentation
- OpenAI — evaluating model outputs — Corrections as labeled examples are the operational version of evaluation data.