A curated walkthrough is not evidence
RuleCaddie's early bet was whether answers could be forced to cite real Rules of Golf numbers. Veto's was whether secrecy could live in Postgres instead of in a UI that only looks private. Both questions can be "answered" by a five-minute walkthrough that never touches the hard case. That walkthrough is theatre. Evidence is what you agreed would count before the artifact started looking finished.
Polish changes perception, not the decision
A prototype earns authority from the evidence around it, not from how finished it looks. AI made that trap cheaper: an afternoon can produce something clickable enough that the conversation slides from "should we build this" to "what would it take to ship what we already have." The artifact started answering a question nobody agreed to ask.
We have watched this with grounding spikes and privacy spikes alike. One clean RuleCaddie answer with a rule number on screen feels like validation. One Veto reveal where nobody's client sees another person's vote feels like proof. Neither is the experiment. The experiment is the fixed set of situations, the attempt to reconstruct a veto from what the client receives, the recovery path you deliberately put in the session.
Kill criteria tell you when to stop. They do not tell you what observation is allowed to move the keep decision. That is a separate line on the brief: what we will watch, who will do it, and which results count as positive, negative, or inconclusive — written before anyone sees the demo.
Evidence has to match the question
A usability claim needs target users attempting the task. An operational claim needs failure, recovery, or load. A claim about agent reliability needs a representative set, not one impressive trace selected for the room. Stakeholder enthusiasm shows the idea is persuasive. It does not validate need, feasibility, or roadmap priority.
For RuleCaddie, the question was citation under pressure, not whether a golf rules chat could look sharp. The evidence was a fixed set of course situations where inventing a number is tempting and refusal is allowed. For Veto, the question was reconstructability from the wire, not whether the UI hid a badge. Showing the happy path and stopping is how you launder a vision artifact as a feasibility result.
If neither a positive nor a negative result would change what you do next, you are not running an experiment. You are producing a demonstration. That can still be useful — alignment, fundraising theatre, a design review — but label it. Do not cite it later as the keep decision.
Name the shortcuts, or they rewrite the conclusion
Shortcuts are what make a prototype cheap: mocked billing, a test tenant, clean data, one language, a single happy path, manual intervention behind an automated-looking flow. They are not a failure. They are a boundary on what you may conclude.
For each shortcut we ask: which conclusion are we no longer allowed to draw because this is fake? A mocked API cannot validate current integration behavior. Internal stakeholders cannot stand in for customer usability. A curated input list cannot stand in for an eval set. When the walkthrough hides those limits, the room fills the gap with hope.
We put the shortcuts on the same card as the question. A forwarded link, screenshot, or recorded demo should still say what the artifact was designed to test and that it is not approved for production or customer promises. Context that only lives in the builder's head expires the moment someone else presents the build.
Close the card when the mandate ends
Give the prototype a time, session, or learning limit. Without an expiry, someone adds another path because the first one looked promising, a stakeholder asks whether it can be shown to a customer, and the team quietly stops learning and starts developing — without the reviews a product owes.
When the limit hits, write a short closeout: result against the predeclared criteria, the decision that changes now, and the artifact status — deleted, archived, fixtures retained, or proposed for an independent production review. Keep authorizes a rewrite, not a promotion of discovery code. That pairs with the kill-criteria habit: stop conditions up front, evidence plan beside them, disposition decided before attachment.
The failure modes are familiar. The sales demo gets cited as validation. The happy-path proof becomes "production ready." The team rewrites the hypothesis after seeing the result. The orphan prototype survives with no owner. Closing the card is how you stop later readers from reinventing what the walkthrough was never allowed to prove.
What we write before the first prompt
Before a consequential prototype starts — especially one an agent can make look finished in an afternoon — the brief has five lines: the one decision the artifact may change, the evidence that would count, the shortcuts that limit that conclusion, when the mandate ends, and what happens to the implementation afterward.
During the run we record observations and newly discovered shortcuts. We do not expand the prototype because implementation is going well. After closeout, a positive result changes only the named decision. A negative result is useful when the test was valid. An inconclusive result means a new card, not a quieter rewrite of the first question.
This sits next to fake-door tests and fidelity ladders without replacing them. A fake door measures demand without building the thing. A fidelity choice matches cost to the question. A curated walkthrough is neither. It is the artifact persuading people before the evidence plan has run.
Questions
- When is a demo allowed to count as evidence?
- When the walkthrough is the observation named on the card — same people, same cases, same pass/fail rules written before the artifact existed. A polished path chosen for the room does not qualify.
- How is this different from kill criteria?
- Kill criteria say when to stop the bet. The evidence plan says which observations are allowed to move a keep or kill. You need both: a stop signal and a matching measurement, written before attachment.
- What if the prototype reveals a better question mid-run?
- Preserve the original card and open a new one. Rewriting the hypothesis after seeing the result is how a walkthrough quietly becomes the decision.
Sources
- The Prototype Is a Question, Not a Product — Decision card: question, evidence, shortcuts, expiry, disposition — polish ≠ evidence
- Proof of Life (PoL) Probe — Falsifiable pass/fail before build; disposal date; one hypothesis
- Prototype Theatre: When Demos Replace Proof — Internal applause and curated demos vs behavioral proof