Human-in-the-loop is a design decision, not a limitation
The review step in the Deerman PO tool is not a temporary apology for weak AI. Outgoing purchase orders are commitments someone must own; automation that removes the last look does not remove accountability. Better models should shrink the queue to genuine ambiguity — not eliminate the human from the loop.
Two ways to read a review queue
The first reading is that a review queue measures how much your automation has failed. Under that framing, every flagged line is a defect, the queue length is a scoreboard, and the goal is zero. Teams that adopt this framing tend to start tuning thresholds to reduce flag volume, which works exactly the way you would expect: fewer flags, more bad orders.
The second reading is that the queue is where uncertainty is supposed to end up. Under that framing, a flagged line is the system working — it identified something it could not confirm and routed it to the one component that can resolve ambiguity, which is a person who knows the customer and the catalog.
These produce different software. The first optimizes for an empty queue. The second optimizes for the queue containing exactly the things that genuinely need a human, and nothing else.
What gets flagged and what does not
A line reaches review when it does not reconcile — the code did not resolve against the manufacturer sources, or the reconciliation produced something inconsistent. That is a mechanical condition, not a judgment call, and it means the flag has a stated reason attached to it rather than a score.
Lines that reconcile cleanly do not go to review, and that is the other half of the design. A review step that shows a person everything is not human-in-the-loop, it is manual data entry with extra steps, and staff will start clicking through it without reading. The value of the queue is entirely dependent on it being short enough that each item still gets attention.
This is also why the recovery passes matter so much to the review experience. Every failure mode we fix in code is a category of line that stops appearing in the queue, which protects the attention available for the lines that remain.
- 01Machine parsesFast, and wrong in known ways
- 02Machine matchesAgainst the real price files
- 03Machine flagsAnything that doesn't reconcile
- 04Human confirmsThe only step that can't be skipped
- 05System sendsNow it's safe to be automatic
Confident enough to do the work, not confident enough to skip the check.
Confidence scores are not an interface
A common alternative is to surface a model confidence number and let the user decide. We avoided that. A number between zero and one tells an operations person nothing actionable — it does not say what the system was unsure about, and it invites the fiction that 0.82 and 0.79 are meaningfully different.
What the review modal shows instead is the concrete problem: this code did not match anything in the manufacturer data, this line's quantity and price look transposed relative to the rest of the order. The user is being asked a question they can actually answer from the document in front of them. That framing also decides what has to be on screen. The flagged line sits next to what the parser read and what the document appears to say, close enough together that the comparison needs no scrolling and no memory. A reviewer forced to hold two values in their head while moving between views will start approving on pattern rather than reading, and a review step that gets pattern-approved is indistinguishable from no review step at all.
The corollary is that a flag has to be specific enough to write a sentence about. If the system cannot say what is wrong with a line, it does not have enough information to usefully involve a human, and the fix belongs upstream in the parsing rather than in the interface.
The accountability argument
The strongest reason for the review step has nothing to do with model quality. An outgoing purchase order is a commitment — a real product, a real price, a real delivery to a customer who will be annoyed in a concrete way if it is wrong. Somebody at the rep firm is answerable for that order.
Automation that removes the last human touch does not remove that accountability, it just removes the moment where it could have been exercised. The person is still answerable; they simply no longer had a chance to look. That is a worse position for them than the extra thirty seconds of review, and no amount of parser improvement changes it.
So the review step survives model improvements. What model improvements change is the volume flowing through it, not whether it exists. Better parsing means the queue holds only genuinely ambiguous documents. It does not mean nobody approves the order.
Designing the loop, not tolerating it
Once you accept the review step as permanent, you build it differently. It gets the same care as the rest of the product rather than being an admin screen bolted on the side. Corrections are stored, not just applied, because they are evidence about what the document said. The reason for each flag is written in plain language. The modal is reachable in one step from the upload, because it is part of the main path, not an exception path.
The reframe is small and it changes the roadmap. Nobody on this project is working toward the release where the human comes out of the loop. The work is making the loop short, specific, and worth the operator's attention.
If you are integrating AI into an operational workflow, decide this early and say it out loud. A review step you have designed is an asset. A review step you are apologizing for tends to get quietly weakened until the day it stops catching anything.
Questions
- Should the goal of a review queue be zero flagged items?
- No. Treating every flag as a defect pushes teams to loosen thresholds until bad orders slip through. Design the queue to hold exactly what cannot be confirmed mechanically, and keep it short enough that each item still gets attention.
- Why avoid showing model confidence scores to operators?
- A number between 0 and 1 is not actionable and invites fake precision between 0.82 and 0.79. Show the concrete problem — unmatched code, transposed quantity and price — beside the document evidence the reviewer can check.
- What changes once review is accepted as permanent?
- You build it as a primary path: plain-language flag reasons, corrections stored as evidence, one step from upload. Roadmap work makes the loop short and specific rather than racing toward a release that removes the human.
Sources
- PO processing tool case study
- NIST — AI Risk Management Framework — Human oversight as a deliberate control, not a temporary patch.
- Google PAIR — People + AI Guidebook