Five demos are not an eval harness
Prompt and model edits get shipped on vibes: five examples in a playground, a couple of screenshots, a feeling that the new wording is sharper. On RuleCaddie and BuilderHelp that is how regressions land — a better happy path that stops citing, stops declining, or invents a rule number. An eval harness is the boring alternative: a versioned golden set, checks you can pin, and a gate that runs before the change merges.
The failure mode five demos miss
The demos are almost always the cases the author was thinking about when they rewrote the prompt. They improve, or at least stay fine. The damage shows up on the cases nobody opened: a local-rules question that used to decline and now invents a ruling, a citation that used to travel with the retrieved passage and now gets paraphrased into a plausible number, an owner-update draft that used to stay in propose mode and now sounds like a filed fact.
Those failures do not look dramatic in a playground. They look fluent. In domains where the user is under time pressure — on a tee box, between an inspection and a homeowner call — fluent-and-wrong is worse than a short refusal, because the person least able to catch the error is the one asking.
So the first job of an eval is not to score "quality" in the abstract. It is to protect the contracts you already decided the product must keep: cite or decline, propose don't commit, stay inside the retrieved text.
What belongs in the golden set
Start from real failures and real product rules, not from a generic benchmark. For RuleCaddie that means questions where the answer must cite a specific rule, questions that must decline because the passages do not cover them, and questions where a confident wrong ruling would cost a stroke. For BuilderHelp it means owner-update drafts that must not invent budget numbers, invoice matches that must stay proposals, and schedule suggestions that must not silently rewrite the calendar.
Each example needs an identity that survives refactors, the input the system actually sees, and an expected shape — not always a single gold string. Sometimes the expected shape is structural: a citation present and matching the retrieved rule id, a decline flag set, a JSON field still null because the human has not confirmed. Deterministic checks on those shapes catch most of the regressions that matter.
Keep the set in the same repo as the prompts, in a format that diffs cleanly in review. Adding an example that the current prompt fails is a deliberate decision; deleting one that it fails is how scores get prettier without the product getting better. Treat both as reviewable changes, not as a quiet edit to a spreadsheet.
Pin the harness, or the scores are noise
A harness that calls a model, optionally calls a judge, and aggregates results is only useful if the same input produces a comparable score tomorrow. That means pinning the application model identifier, the judge model snapshot if you use one, temperature at zero for classification-style checks, and the judge prompt itself as a versioned file. An unpinned "latest" judge is a measuring stick that changes length between runs.
Prefer hard assertions before soft scores. "Response includes rule 16.1b" and "decline is true when retrieval is empty" are checks a CI job can fail loudly. LLM-as-judge scores are a second tier for prose quality once those contracts hold. If you only have the soft score, every prompt rewrite becomes an argument about whether 0.84 is still good enough.
Log the metadata with the result: app commit, dataset commit, prompt file hash, model ids. Without that, you know quality moved and you cannot say which lever moved it.
Gate the three changes that actually shift answers
Not every pull request needs an eval run. Three classes of change do: prompt text and few-shot examples, model or inference configuration, and retrieval settings for anything grounded — chunking, embedding model, reranker, similarity thresholds. Those are the edits that change what the model sees or how it is instructed. Path-filter the CI job to those files so the suite stays fast enough that people leave it on.
Gate on deltas and on blocking contracts, not on a single vanity threshold. A citation-miss rate that jumps from 1% to 6% should block. A stable 4% fail rate the team already accepted at launch might only warn. Absolute thresholds alone get loosened until they never fire, which is how the gate dies in week two.
When the gate fails, the PR comment should name the failing example ids. An aggregate score without the cases is a reason to shrug; a short list of broken contracts is a reason to fix the prompt.
Grow the set from production, not from imagination
The golden set is unfinished on day one. The useful growth path is the same one we already use for corrections telemetry: when a user fixes a parse, rejects a draft, or reports a bad ruling, format that case into an example with the expected shape and open a small PR. That turns a one-off incident into a permanent check.
Resist the habit of updating ground truth to match whatever the new prompt emits. If the product rule changed, update the expectation and say so. If the model simply drifted, the failing example is doing its job.
The practical bar for shipping a prompt change is not that five demos look nicer. It is that the contracts still hold on the set you have, and that any new failure mode you just learned about is about to join that set.