Shadow mode is not a soft launch
When a team wants an agent in production without the risk of production writes, the instinct is a soft launch: few users, quiet channel, full tool list. That is not shadow mode. Soft launch still lets the model mutate. Shadow mode runs on real work with read-only or propose-only tools, logs what the agent would have done, and graduates write access by action class when the evidence holds — BuilderHelp invoice matching, RuleCaddie answers, Deerman PO drafts included.
Two different safety stories
A soft launch answers who can see the feature. A small cohort gets the full product, including tools that create, update, post, or send. If the model is wrong, the blast radius is smaller, but the failure mode is unchanged: a bad tool call still lands in a live system.
Shadow mode answers what the agent is allowed to do. The agent sees the same inputs the human sees — an invoice batch, a rules question, a PO draft — and produces a structured proposal. Nothing mutates until a later, separate promotion. The safety story is permission, not audience size.
Teams blur them because both feel cautious. They are not interchangeable. Shipping write tools to ten users still teaches the agent that writes are available. Running propose-only on every real ticket teaches you whether the proposals are worth promoting.
What shadow mode has to produce
Shadow is not a logging afterthought on a write-capable agent. The tool surface itself has to be propose-only: read, retrieve, draft, score — not post, not send, not update the ledger. The proposal should use the same schema you will later put behind an approval gate, so shadow data is comparable to gated data.
You also need a join to what a human actually did. On BuilderHelp that might be the invoice match the AP clerk accepted; on RuleCaddie, the citation set they kept; on a PO flow, the lines they approved. Without that ground truth, you have a diary of suggestions, not a promotion signal.
Offline golden sets still matter — five demos are not an eval harness — but shadow is the online counterpart. It catches the long tail that never made it into the fixture file: vendor layouts you have not seen, rules questions phrased sideways, tickets that resolve themselves.
Promote by action class, not by vibes
The unit of graduation is a narrow action class, not "the agent." Matching a vendor remittance line is not the same as posting an AP entry. Citing a rule number is not the same as filing a scorecard change. Each class earns write access — or more often, an approval gate — on its own agreement rate against human outcomes.
Promote the smallest class first. One reversible, low-blast action with a clear inverse. Leave everything else in propose-only. A global readiness score invites the failure mode where the happy path looks solid and the dangerous tool was never exercised in shadow.
Promotion changes the mode, not the leash overnight. Shadow graduating usually means the proposal now waits for a human click. Full autonomy, if it ever arrives, is a later promotion with its own evidence from approval outcomes and reversions — not a reward for surviving a soft launch.
How this sits next to human-in-the-loop
Human-in-the-loop is about accountability after a system can act: who owns the PO before it ships, what stays in the review queue even when the model is confident. Shadow mode is about what you refuse to wire until you have evidence. One is product design for irreversible commitments. The other is a rollout discipline for tool permissions.
They compose. Shadow produces the proposals. An approval gate is often the first write path you enable. Corrections and reversions become the telemetry that promotes or demotes a class. None of that requires giving the model day-one write credentials "just for the pilot cohort."
The practical rule we use in scoping: name the first ship as propose-only on real traffic, name the action classes that might later write, and name what evidence moves each class. If the plan starts with write tools behind a feature flag for friends and family, rewrite it — that is a soft launch wearing a safety label.
What we check before enabling a write tool
Read and propose tools are live on the real workload, not only on a demo tenant. Proposals share the schema the approval path will use. There is a join to human actions for the classes under review. Agreement and edit/reversion rates are tracked per class, not as one agent-wide number.
Unknown or out-of-policy proposals demote to human review instead of executing or hard-failing into silence. Credentials behind any future write tool are scoped to that action class and environment — a "restart" tool backed by admin rights is not least privilege with a friendlier name.
If those checks are missing, keep the model in shadow. A quieter launch with write access is still write access. The milestone that reduces risk is propose-only evidence on real work, not a smaller audience for the same mutations.
Questions
- Is a soft launch the same as shadow mode?
- No. A soft launch limits who sees the feature; write tools can still mutate live systems. Shadow mode limits what the agent may do — real traffic, propose-only tools, and promotion by action class against human outcomes.
- When should write tools come online?
- After a narrow action class shows stable agreement with human decisions in shadow, usually behind an approval gate first. Full autonomy is a later, separate promotion with its own reversion evidence — not the default next step after a pilot cohort.
- How does this relate to human-in-the-loop?
- Human-in-the-loop designs accountability for irreversible actions. Shadow mode is the rollout discipline that keeps write credentials off until propose-only evidence exists. They stack: shadow proposals, then gated writes, then corrections as promotion telemetry.