Copy a summary
A plain-text summary of the verdict for Slack or the pull request.
A verdict on every release
Did this release cost you sign-ups, sales or onboarding? Every release is compared with the one before it, from getting in to getting things done in your app. You get one plain answer, what the drop costs a month, and who it hit: which browser, device or accessibility setting.
Releasesacme.com · last 90 days
| Release | Biggest change | Verdict |
|---|---|---|
| 2.5.2 · 2 hours ago | About 4 hours more at the current pace | Too early to tell |
| 2.5.1 · yesterday | “Sign up” back to 51% | No drop |
| 2.5.0 · 3 days ago | “Sign up” 52% → 38%. Safari users 48% → 7% | Drop found |
| 2.4.3 · 2 weeks ago | “Finished onboarding” 64% → 52% | Early signal |
| 2.4.2 · 3 weeks ago | Nothing fell past its limit | No drop |
2.5.0 dropped “Sign up”, and Safari users took the hit: 7% of them got through, down from 48%. About £4.8k a month at risk. 2.5.1 fixed it.
From a deploy to a verdict
Send each deploy
Call the deploy hook from CI, point GitHub, Vercel or Netlify at it, or let new app versions in your events mark releases on their own. You can add one by hand too.
Every release gets a verdict
Before against after, for getting in and for getting things done, and for every accessibility setting, browser and device. No drop, Early signal, Drop found or Too early to tell.
Fix it, and see it recover
Open a case for the group it hit, jump to the friction and replays from the release window, and watch the next release’s verdict for the recovery.
Compared across the whole journey, not just checkout
A release can let people in and then stop them getting anything done. So each release is checked twice: did as many people get in, and did they still finish what they came to do?
- Getting in: each conversion goal’s rate, the share of visitors who converted, before and after the release.
- Getting things done: each protected flow, like finishing onboarding, inviting a teammate or checking out. Success counts only completions that weren’t taken back by a refund, cancel or problem report within seven days.
- Past the sale: sign-up, then onboarding, then the first project. A drop after conversion costs you just as surely as one before it.
Release 2.5.0compared with 2.4.3
VerdictDrop found
At risk£4.8k/month
Checks5
| Check | Before | After | Verdict |
|---|---|---|---|
| Getting in: Started a trial | 9.8% | 9.5% | No drop |
| Getting in: Bought a plan | 3.2% | 2.4% | Drop found |
| Getting things done: Sign up | 52% | 38% | Drop found |
| Getting things done: Finished onboarding | 61% | 60% | No drop |
| Getting things done: Invited a teammate | 34% | — | Too early to tell |
The figure is the biggest single drop, not a sum: “Bought a plan” and “Sign up” lose the same customers, so they aren’t counted twice. “Invited a teammate” needs about 9 hours more at the current pace.
What the drop costs, a month
When you track revenue, a drop is valued at the current pace, using what a completion was worth before the release. The release’s figure is its largest single drop, not a sum, so the same lost customer is never counted twice.
- About £4.8k a month at risk reads as it sounds: if the drop holds, that’s what it costs.
- No revenue tracked, no made-up money. The verdict still says what dropped and for whom.
Who it hit
A release that breaks one browser, or a keyboard path, can hide inside a small overall dip. For each protected flow, every group is judged after the release: accessibility settings (keyboard, zoom, large text, reduced motion, high contrast and forced colors, plus screen readers in your iOS, Android, React Native and Flutter apps), browsers, operating systems and device types.
A group counts against the release only when it kept up before, is behind now, and its own fall since the release is past the limit and clear of noise. Gaps that were already there are listed separately, as not blamed on this release.
Who this release left behindSign up · release 2.5.0
| Group | Before | Now | Verdict |
|---|---|---|---|
| Safari users | 48% | 7% | Drop found |
| Large text | 50% | 37% | Early signal |
| Forced colors | 31% | 30% | Already behind before |
Forced colors was already behind before 2.5.0, so it’s listed under “not blamed on this release”. It’s still a gap worth fixing, just not this release’s doing.
Browsers don’t say whether a screen reader is running, so on the web keyboard use stands in for it.
Red only when it’s real
A release verdict your team learns to ignore is worse than none. So every drop has to clear a limit and a test before it counts.
- Getting in drops when a goal’s rate falls by a fifth or more of what it was, with at least 100 visitors on each side.
- Getting things done drops when a protected flow falls by more than its limit, 10 points by default, with attempts from at least 15 different people on each side.
- Tested together, so the chance of any false “Drop found” in a release stays under 2.5%. Past the limit but not yet clear of noise is an Early signal, never green.
- Too early to tell comes with an estimate, like “about 4 hours more at the current pace”.
- Fair windows: before runs from the previous release (or a week earlier, whichever is later). After runs to the next release, now, or two weeks in. A report settles seven days after its window, once refunds can no longer take completions back.
From verdict to fix, in one place
Each report comes with the next steps already lined up.
Copy a prompt
A ready-made prompt for your AI assistant, pointing it at this release’s report.
Open a case
One click opens a case for the group the release hit, with the step and what it cost.
Friction and replays
Jump to friction and replays, already filtered to the release window.
Hear about it, or gate the release in CI
When a release’s verdict becomes Drop found, an alert goes out once: by email, and to Slack, Teams or a webhook if you’ve set them up. You can also be told each release’s first verdict when it’s No drop or Early signal.
In CI, a step after deploy can ask for the verdict as JSON, wait until it’s more than too early to tell, post it, and fail the build on a drop if you want a release gate.
- Webhook events:
alert.release_droppedandrelease.verdict. See the webhook reference. - CI verdict endpoint: uses your app’s deploy hook token. See wait for a release verdict in CI.
# Ask for the verdict on the release you just shipped
curl -s "https://in.metrickle.com/v1/releases/$HOOK_TOKEN/verdict?version=2.5.0"
{
"verdict": "dropped",
"sentence": "Release 2.5.0 dropped “Sign up”: 52% → 38%.",
"lostPerMonth": 4800,
"report": "https://app.metrickle.com/apps/…/releases/…",
…
}Ask your AI assistant what the release broke
Over MCP, assistants like Claude get list_release_verdicts (every recent release and its verdict) and get_release_report (the full before and after, with the groups it hit). Ask “did yesterday’s release hurt onboarding?” and get the answer with the numbers.
Common questions
What is a release compared on?
The release before it. Each conversion goal’s rate (getting in) and each protected flow’s success (getting things done) are measured before and after the release. Before runs from the previous release, or a week earlier if that’s later. After runs to the next release, now, or two weeks in. For each protected flow, every accessibility setting, browser, operating system and device type is judged too.
Can one bad session fail a release?
No. A protected flow needs attempts from 15 different people on each side, and a goal 100 visitors, before either is judged. A drop must also pass its limit and be clear of noise, with every comparison in the release tested together so the chance of any false “Drop found” stays under 2.5%. Until then it’s an Early signal or Too early to tell.
What does “Too early to tell” mean?
Not enough people have used the release yet to say either way. The report estimates how long until there are, like “about 4 hours more at the current pace”.
Where do releases come from?
A deploy hook you call from CI, GitHub, Vercel or Netlify webhooks pointed at the same URL, releases added by hand, or new app versions seen in your events, which suits iOS, Android and Flutter apps.
How is the money worked out?
Only when you track revenue. A drop is valued per month at the current pace, using what a completion was worth before the release. The release’s figure is its largest single drop, not a sum.
When is a verdict final?
Seven days after the release’s window ends, because a refund, cancel or problem report can still take a completion back until then.
See where your apps break, and who they break for
Free to start. One script tag on the web, one SDK on mobile, and no cookie banner in cookieless mode.
Start free