A judge score is not an evidence path
When an agent looks right in an eval dashboard, the first number people trust is often an LLM-as-judge score — groundedness, helpfulness, faithfulness on a 0–1 scale. That number can be high while the trace shows the agent never retrieved the artifact the answer depends on. A judge reads the final text. An evidence path check asks whether the tools, searches, and citations the agent actually ran were allowed and sufficient for the claim. Those are different tests. Treating the soft score as proof of grounding is how RuleCaddie citations and BuilderHelp invoice matches pass a review that would fail a contract.
What a high judge score can miss
A judge model is good at reading fluent prose. It is weak at proving process. If the answer cites a rule number that sounds right, or summarizes an invoice match in clean language, the judge often rewards the surface. It does not independently verify that retrieval returned that rule passage, that the match tool was called with the vendor remittance lines in scope, or that the agent searched the places where absence must be proven before saying nothing was found.
The failure mode has a name in recent agent eval work: state-invalid correctness. The final answer may match the expected label, or look attributable to a human reader, while the path violates the task state — wrong permission cone, artifact after the as-of time, citation to something never fetched, or a "not found" claim after searching only one of the required stores.
That is not a rare corner case on agent workloads. It is the blind spot final-answer and judge-only evaluation have by construction. The dashboard stays green. The product still lied about how it knew.
Score the path with a contract
An evidence path check is a machine-checkable contract over the recorded trajectory. For each example you name what must have happened before a claim is allowed: which tool classes may run, which stores must be searched before absence, which artifact ids may be cited, and which fields in the structured answer are required. The scorer compares the trace to that contract. Pass and fail are reproducible without a second model.
On RuleCaddie that looks like: retrieval ran against the Rules of Golf corpus, the cited rule id appears in the retrieved set, and decline is true when the set is empty. On BuilderHelp invoice matching: the match tool saw the remittance lines, the proposal schema stayed propose-only, and no ledger write appears in the trace. Those are assertions a CI job can fail loudly.
Keep the contract next to the golden set — same repo, same reviewable PR. When you change the product rule, update the contract and say so. When the agent drifts, the failing example id is the signal, not an argument about whether 0.84 is still good enough.
Judges stay useful — after the contracts hold
None of this says throw out LLM-as-judge. Soft scores still help with tone, clarity, and whether a proposal reads like something an AP clerk would keep. They are a second tier. Hard assertions on cite, decline, schema, tool order, and forbidden writes come first. If only the soft score exists, every prompt rewrite becomes a debate about thresholds.
Pin both layers. Application model id, judge model snapshot, temperature for classification-style checks, and the judge prompt as a versioned file. An unpinned "latest" judge is a measuring stick that changes length between runs — the same discipline we already use for golden-set harnesses.
When a judge and a trajectory contract disagree, believe the contract. A 0.9 groundedness score on an answer whose evidence id never appears in the fetch log is not a mystery. It is the judge doing prose evaluation on invalid process.
Grow the suite from invalid paths, not prettier answers
Add examples when you catch a fluent miss: a citation that was never retrieved, a decline that should have fired, a PO draft that sounded filed, a "no match" after searching the wrong vendor folder. Format the input, the expected path shape, and the expected answer shape into the set. That turns an incident into a permanent gate.
Resist updating ground truth to match whatever the new prompt emits. If the product rule changed — for example, a new store that must be searched before absence — update the contract and document why. If the model simply skipped the tool, the failing example is doing its job.
The practical bar for shipping an agent change is not that the judge average moved. It is that the evidence path contracts still hold on the set you have, and that any new invalid path you just learned about is about to join that set.
Questions
- Is an LLM-as-judge score enough for agent eval?
- No. Judges score final prose. Agents also need deterministic checks on what they searched, fetched, cited, and were allowed to access. Soft scores are useful after those contracts hold.
- How is this different from a golden-set harness?
- A golden set pins inputs and expected answer shapes. An evidence path contract pins the trajectory those answers must come from — tools, retrieval ids, search coverage, and permission or time bounds. You need both.
- What should block a merge?
- Failing path contracts — missing retrieval for a citation, decline not set when the set is empty, a write tool appearing in a propose-only example. Judge-score deltas can warn; path failures should block.