An experiment flag is not a release flag
Release flags and experiment flags share an SDK and a dashboard. They do not share a job. A release flag hides unfinished code until a cohort is ready, then the dead path should die. An experiment flag assigns people to variants so you can answer a named question, then the losers should die. On BuilderHelp we have watched teams ramp a "test" with no hypothesis, and leave an A/B harness up for months after the winner was obvious — same boolean, wrong lifecycle. Type the flag when you create it, or the inventory will quietly invent a third product nobody meant to maintain.
Same boolean, different job
A release flag hides unfinished work until a cohort is ready. The success condition is boring: everyone who should see the feature sees it, the old path can die, and the flag leaves the inventory. An experiment flag routes people into variants so you can measure a difference. The success condition is a closed question — which variant won, by what metric, with sticky assignment for the window — then the losing paths go away.
In the SDK they look identical: a key, a boolean or a string, a targeting rule. That is why teams reuse one for the other. A release ramp that never had a hypothesis becomes an accidental A/B test with no sample size. An experiment that "won" keeps evaluating forever because removing the flag feels like deleting the feature.
On BuilderHelp we name the type when the branch lands. Release flags own rollout clocks across web, Expo, and the OCR worker. Experiment flags own assignment stability and an analysis window. Mixing those jobs is how you get a dashboard full of toggles and no one who can say whether the feature shipped or the test is still running.
Release wants a finish line. Experiment wants a verdict.
Release work asks: is the new path safe enough to be the only path? You widen exposure, watch error and latency budgets, keep a kill path while trust builds, then delete the dead branch. The percentage ramp is a delivery tool, not a science experiment. You are not proving conversion; you are proving the build holds under load.
Experiment work asks: does variant B change the metric we named before traffic moved? Assignment must stick for the window. Changing the split mid-flight, flipping winners by gut feel, or treating a 10% ramp as "the test" without a stopping rule turns the flag into theater. When the window closes, you pick a winner in writing — then the code becomes that winner, not a permanent lottery.
If you cannot state the metric and the stop date on the PR that adds the flag, you are not running an experiment. You are shipping under a release flag and should call it that. The cleanup clock is shorter and the owner is responsible for deletion, not for a stats readout nobody scheduled.
What breaks when you conflate them
Product keeps "optimizing" a release that was already the intended ship. Engineering keeps two implementations because the experiment never recorded a winner. Support cannot answer whether a customer is on the new flow or still in a cohort. Analytics reports a lift that is really a partial rollout against a stale control.
Ops gets the worst of both. Someone treats an experiment variant as a kill switch and flips traffic during an incident without knowing which path is the stable default. Someone else refuses to delete a release flag "in case we want to A/B later," so the temporary fork becomes the platform.
We have already written that shipping under a flag is not finishing the flag, and that a Next.js-only gate is not a multi-surface rollout. Type confusion is the sibling failure: the inventory looks healthy because percentages move, while the decision model — ship versus measure — never happened.
How we label and retire each type
Release flags carry owner, surfaces that still evaluate, target removal, and an abort story. When exposure hits the intended cohort and stays clean, the removal PR deletes the branch. Archiving the config waits until every still-shipping client stopped reading the key.
Experiment flags carry owner, hypothesis, primary metric, assignment rule, start and stop dates, and who will write the verdict. Mid-window targeting changes need the same seriousness as changing the experiment definition. At stop, the verdict lands in the decision log — Accepted winner, Rejected variant — then code collapses to one path on the same clock discipline we use for release cleanup.
Operational kill switches stay a third type. They are not experiments and they are not unfinished releases. If a release or experiment keeps getting extended "as a safety net," reclassify it explicitly rather than lying with a temporary label. Extending silently is how temporary becomes forever.
What we check in the inventory
Open the flag list and sort by type. Every experiment past its stop date without a written verdict is unfinished research wearing a production toggle. Every release at 100% past its removal date is unfinished shipping. Count those before you add another key.
For each live experiment, confirm sticky assignment and that the control still exists on purpose. For each live release, confirm the abort path and which surfaces still evaluate. If the same key is doing both jobs, split it or pick one job and delete the other story from the runbook.
The milestone is a small inventory where each flag has one verb: ship, measure, or operate. When the verb is done, the branch leaves. That is the whole discipline — not a larger flag platform.
Questions
- Can one flag start as a release and become an experiment?
- Only if you reclassify it in writing: new hypothesis, metric, stop date, and sticky assignment. Do not silently reuse a ramp as a test — you will read delivery noise as product signal.
- When should an experiment flag be deleted?
- After the stop date and a recorded winner (or a recorded kill). Collapse the code to one path, then archive the key once no shipping client still evaluates it.
- How is this different from finishing a release flag?
- Finishing a release means the intended path is the only path. Finishing an experiment means you chose that path from measured variants. Both end in deletion; only one required a verdict.
Sources
- Feature Toggles (aka Feature Flags) — Martin Fowler / Pete Hodgson — Canonical taxonomy: release, experiment, ops, permissioning — different lifespans.
- Feature Flag Rollout Strategies 2026 — Toggle type vs evaluation tier; conflating types as a root of flag debt.
- Related: Shipping under a flag is not finishing the flag
- Related: A Next.js feature flag is not a rollout