A feature matrix is not a bake-off
When a client or studio asks us to help pick a CMS, a payments provider, or an OCR vendor, the first artifact that usually appears is a feature matrix. Rows of capabilities, columns of vendors, cells filled with checkmarks from marketing pages. It feels rigorous. It almost never settles the decision. The work that does is smaller, shorter, and written down before anyone has forgotten why they said no.
What a matrix quietly optimizes for
A feature matrix rewards breadth. The vendor with the longest list looks safest, and the person filling the sheet can finish without touching the product. That is useful for eliminating tools that clearly cannot do the job. It is a poor way to choose among three that all can.
The cells also tend to answer the wrong question. "Supports preview" is not the same as "an editor can preview this site's draft homepage without a deploy." "Has webhooks" is not the same as "our order ingest stays idempotent when the webhook fires twice." The matrix records slogans. The bake-off has to record friction.
We still write one. We keep it short, and we treat any cell we have not exercised as unknown rather than as a checkmark copied from a comparison page.
The bake-off is one realistic path
The useful evaluation is a thin path through the real work. For a headless CMS that usually means one content type that matches how the site is actually structured, ten sample documents from the client's existing copy, one preview flow into the front end they will ship, and one editing session with the person who will publish after launch.
We have run that shape when choosing Sanity for content-heavy marketing sites, including our own migration off WordPress and builds like Provale Cup where clinicians need to change product explanation without waiting on a deploy. The question was never whether Sanity or Contentful can store fields. It was whether the editor could change a singleton page, see the draft, and trust the publish button on day two.
For other vendors the path changes, but the rule does not. A payments bake-off exercises one real charge, one failure, and one refund against the account model you already have. An OCR bake-off runs the client's messy documents, not the vendor's clean sample PDFs. If the path does not include the ugly part of the job, you have not evaluated the product.
- Eliminates obvious non-fits
- Copies vendor marketing language
- Leaves editor friction untested
- Easy to finish without building
- Exercises one real path end to end
- Uses the client's content and failure cases
- Surfaces preview, auth, and ops seams
- Ends with a written reject/accept log
The matrix narrows the field. The bake-off is what makes the choice defensible six months later.
Put the people who will live with it in the room
Developers alone will optimize for schema flexibility and local tooling. Editors alone will optimize for the friendliest blank canvas. Neither view is wrong, and either one alone will pick a tool the other group quietly resists.
So the bake-off includes both, on a timer. An engineer models the content and wires the preview. An editor tries to publish something real and narrates where they hesitate. The hesitation is the data. A polished vendor demo never produces it, because the salesperson is driving.
When we skip that session, we learn the same facts later as support tickets. The cost is identical. The timing is worse.
Write the decision log the same day
The bake-off is not finished when someone says they have a preference. It is finished when a short log exists: what we chose, what we rejected, the two or three scenarios that decided it, the assumptions that still have to hold, and who owns each open condition.
We write it the same day on purpose. A week later the room remembers the winner and forgets the runner-up's fatal friction. That forgetting is how teams reopen settled decisions the first time a blog post praises the tool they did not pick.
Dissent belongs in the log too. If someone preferred the other option, name the concern and how it was answered — mitigated, sequenced, or explicitly accepted. An undocumented minority opinion does not go away. It waits for the first rough week in production and then restarts the bake-off from memory.
Treat conditions as gates, not reminders
Most selections end with soft conditions: "fine if SSO lands before launch," "fine if the export API covers our model," "fine if legal signs off on the DPA." Soft conditions become unconditional exposure the moment implementation starts and nobody checks the calendar.
We put due dates and owners on those lines, and we treat a missed date as a decision point. Accept the risk in writing, renegotiate scope, or reopen the alternative. What we do not do is let a conditional yes age into a silent yes while the team builds against it.
This is the same discipline we use on kill criteria for prototypes and on support arrangements behind a design studio. The decision only stays healthy if the unfinished parts stay visible.
When a bake-off is the wrong tool
If one option is already load-bearing in the stack and working, a bake-off is theater. Switching CMS because a matrix preferred a rival is how you buy a migration you did not need. Evaluate the gap you actually have, not the category in the abstract.
It is also the wrong tool when the team cannot name a binding constraint. Without a constraint — editor workflow, offline capture, idempotent webhooks, a specific integration seam — every demo looks fine and the matrix fills with optimism. Spend the week clarifying the job before you rent two sandboxes.
Used well, the bake-off is short on purpose. A few days of realistic path-testing beats six weeks of slide comparison, and the decision log is what keeps the choice from being re-litigated every time a new comparison post lands in someone's inbox.