A crash-free rate is not a ship gate
A TestFlight build with a high crash-free session rate feels like permission to ship. It is not. The percentage is a useful early signal that the build is not falling over for everyone. It does not prove you can read the crashes you do have, that hangs and memory kills were checked, or that a long session on the oldest supported device still holds up.
What the rate is allowed to prove
Crash-free sessions answer one question: of the sessions that reported in, what share exited without a crash report. Industry dashboards and App Store Connect stability views make that number easy to watch, and teams correctly get nervous when it drops. Watching it is fine. Treating a quiet week as a done criteria is not.
The rate is blind to sampling. A small TestFlight group that opens the app for two minutes after install can look healthier than a production cohort that stays in a handwriting or camera flow for half an hour. It is also blind to severity mix: one crash on the main path for a minority of devices can sit next to a high percentage and still be a release blocker.
We use the number the way we use a smoke test. If it is bad, stop. If it is good, keep going through the rest of the gate. Good is not finished.
Unsymbolicated stacks look like progress
Xcode Organizer and TestFlight crash feedback only help if the build's dSYM is present and matched to the binary UUID. Without that, you get hex addresses, a tidy crash count, and no actionable top frame. A team can clear a "zero readable blockers" checklist while the Organizer is full of noise nobody can jump into.
The fix is procedural, not heroic. Archive once, upload symbols with the build, confirm the archive still has Download Debug Symbols available, and refuse to triage a release candidate whose top groups are still raw addresses. If MetricKit diagnostics arrive on next launch, the same rule applies: a payload you cannot map to source is telemetry, not a closed investigation.
On Endeo and the BuilderHelp field app we have wasted cycles chasing "new" crash groups that were the same root cause under a rebuild whose symbols never landed. The ship question is not "did crashes arrive." It is "can we name the frame and reproduce it on the reported OS."
Hangs and jetsam are not crashes
A main-thread hang long enough to feel broken often never increments the crash counter. MetricKit hang diagnostics and Organizer hang rate exist because users abandon sessions that never produced a crash report. Jetsam and thermal kills sit in the same blind spot: the process is gone, the session metric may look fine, and the person using the app experienced a force-quit.
Release criteria that only mention crash-free percentage miss that class on purpose or by accident. We treat hang rate and memory-pressure exits as first-class siblings of crashes for any build that does on-device OCR, camera capture, or long Pencil sessions — the paths where main-thread decode and sustained CPU show up after the happy-path tap test is already green.
If your dashboard cannot show hangs separately, you do not have a stability gate. You have a crash gate wearing a stability costume.
Short sessions hide the long ones
TestFlight installs and brief smoke sessions bias toward cold-start health. That overlaps with a trap we already write down elsewhere: a cold start is not a session. The crash-free rate inherits the same bias when most reported sessions are short.
Before we call a candidate shippable we still run a sustained pass on at least one mid-tier or oldest-supported physical device: the core loop for tens of minutes, thermal state watched, not just FPS for thirty seconds on a plugged-in phone. A build can keep a clean crash-free percentage in a quiet TestFlight week and still stutter into serious thermal state once someone actually works in it.
Match the build type too. A debug install on a desk device is not the TestFlight binary. Store-ready configuration, the same entitlements, and the device and OS from the top crash group are the reproduction defaults — not the simulator and not yesterday's local scheme.
What we treat as a ship gate
The rate stays on the dashboard. The gate is a short checklist that the rate cannot satisfy alone: top crash groups symbolicated and owned, hangs reviewed, oldest-supported device session run on the release candidate, and any crash that hits the primary task reproduced or explicitly waived with a reason.
Waivers are allowed. Silent ones are not. "We will watch it in production" is only a decision if someone owns the watch and the rollback. A green percentage with unread stacks and no session test is not a waiver — it is skipping the gate.
Ship when the build survives that list. Celebrate the crash-free number afterward if you want. Do not let it sign the release by itself.
Questions
- Is a high TestFlight crash-free session rate enough to ship?
- No. Use it as a stop-ship signal when it drops, not as proof the build is ready. You still need symbolicated top crashes, hang review, and a sustained session on a real device that matches your support floor.
- What usually makes Organizer crashes useless during triage?
- Missing or mismatched dSYMs. Confirm symbols uploaded with the archive and that top groups resolve to named frames before you treat "no clear blocker" as a real status.
- Why check hangs if crashes look fine?
- Long main-thread hangs and memory kills often never increment crash-free session stats, but users still abandon the flow. For OCR, camera, and long stylus sessions they are as release-relevant as crashes.
Sources
- App Store Connect — View build status and metrics (TestFlight)
- MetricKit — MXMetricManager
- Analyzing crash reports with Xcode Organizer — Symbolication and TestFlight crash triage in Organizer.
- InfoQ — Sustained iOS performance on real devices — Session-length thermal and device-matrix testing beyond cold-start benchmarks.