Release verdicts: did a release fail?
Last updated
Every release gets a verdict that answers one question: did this release fail, and for whom? Metrickle compares each release with the one before it, for getting in (your conversion goals) and for getting things done (your protected flows). It also checks whether any accessibility setting, browser or device fell behind because of it.
Open your app and select Releases. The page lists the releases from the last 90 days, newest first, each with its verdict and its biggest change. Select a release to open its report.
Where releases come from
A release is recorded in any of these ways. See releases to set them up.
- The deploy hook, called from CI, or from a GitHub, Vercel or Netlify webhook.
- A release you add by hand.
- A new app version seen in your events, picked up automatically.
The four verdicts
| Verdict | What it means |
|---|---|
| No drop | Nothing fell by more than its limit, and no group fell behind because of this release. |
| Early signal | Something fell past its limit, but it could still be noise. It's never shown as green. More usage will settle it. |
| Drop found | Something fell past its limit and it's clear of noise. |
| Too early to tell | Not enough people have used the release yet to say either way. The report estimates how long that will take, like "about 4 hours more at the current pace". |
Each verdict is shown with its word and a symbol, never by colour alone. A release's verdict is the worst of its checks.
What a release is compared on
Each report has two parts.
- Getting in. Each conversion goal's rate: visitors who converted, out of all visitors, before and after the release. A goal counts as dropped when its rate falls by a fifth or more of what it was, for example from 5% to 4%. It needs at least 100 visitors on each side before it's judged.
- Getting things done. Each protected flow, like finishing onboarding, inviting a teammate or checking out. Success counts only completions that weren't taken back by a refund, a cancel or a problem report within 7 days. A flow counts as dropped when its success falls by more than its limit, 10 points by default. It needs attempts from at least 15 different people on each side.
Because protected flows can be anything people do in your app, a release is checked past the point where someone signs up or pays, not just up to it.
Before and after
- Before runs from the previous release to this one. If the previous release was more than a week earlier, it starts a week before this one instead.
- After runs from this release to the next one, to now, or to two weeks after the release, whichever comes first.
False alarms
Every comparison in a release, every goal, every flow and every group, is tested together. That keeps the chance of any false Drop found in a release under 2.5%. A drop that's past its limit but not yet clear of noise shows as Early signal.
Who it hit
For each protected flow, the report judges every group after the release:
- Accessibility settings: keyboard, zoom, large text, reduced motion, high contrast and forced colors. In iOS, Android, React Native and Flutter apps, screen readers too. Browsers don't say whether a screen reader is running, so on the web keyboard use stands in for it.
- Browsers, operating systems and device types.
A group counts against the release only when all three are true:
- It kept up before the release.
- It's behind now.
- Its own fall since the release is past the limit and clear of noise.
Groups that were already behind before the release are listed under Other gaps, not blamed on this release. They're still worth fixing, but this release didn't cause them. A group with too little usage before the release to tell is marked that way.
What it costs
When you track revenue, a drop is valued per month at the current pace. It uses what a completion was worth before the release.
The release's figure is its largest single drop, not a sum. A checkout flow and a purchase goal often lose the same customers, so adding them up would count the same money twice. With no revenue tracked, the report still says what dropped and for whom, without a money figure.
When a report is final
A report keeps updating while its after window is open. Once the window closes, its completions can still be taken back by a refund, cancel or problem report for 7 days, so the report settles 7 days after the window ends. Until then it says that rates may still fall a little.
What you can do from a report
- Copy summary: a plain-text summary to paste into Slack or a pull request.
- Prompt for your assistant: a prompt that points your AI assistant at this release's report. Assistants connected over MCP can use
list_release_verdictsandget_release_report. - Open a case: opens a case for the group the release hit. See cases.
- Friction and Replays since release: open the friction report and replays, filtered to the release window.
Which change caused it
For a check that dropped, Show evidence and next steps has a Which change caused it panel. Connect the app's code repository first: Integrations → Releases → Code repository, using a GitHub account from Issue trackers. The token needs read access to the repository's contents.
Find the change compares the release with the one before it in that repository. Releases from a GitHub deployment, Vercel or Netlify carry their commit, so the comparison is exact; otherwise the versions are matched to tags. It then ranks the changed files by how they touch the step where people stop, the element they hit there, the task, and the people who fell behind. For example, a change to keyboard focus ranks higher when keyboard users fell behind, and a change Safari handles differently ranks higher when Safari users did. Tests, docs and lockfiles never rank. Each file shows why it ranked, the matching changed lines and the commit that made them.
If you allow it when connecting the repository, AI also reads the few highest-ranked changes and names the likely cause in plain words, with the smallest fix. It must quote a changed line, and Metrickle checks that line is really in the change. If it isn't, the quote is dropped and the cause is only ever "possible". Only those few diffs are sent to Anthropic's Claude, and only the explanation is kept. Read the change before acting on it: the ranking is a lead, not a verdict.
Alerts and CI
When a release's verdict becomes Drop found, an alert is sent once. It's emailed to you by default, and you can send it to Slack, Teams or a webhook too. You can also choose to hear each release's first verdict when it's No drop or Early signal. Set this up on the Integrations page. See integrations.
To wait for the verdict in your CI pipeline, and fail the build on a drop if you want to, see wait for a release verdict in CI.