Wait for a release verdict in CI

Every release gets a verdict: did it fail, and for whom? (See release verdicts for how it's judged.) The verdict endpoint lets a CI step after deploy wait for that verdict, post it where your team will see it, and, if you want a release gate, fail the job when the release dropped something.

It uses the same token as the deploy hook, so a pipeline that already records releases needs no new secret.

Request

GET https://in.metrickle.com/v1/releases/<deploy-hook-token>/verdict?version=1.4.0

<deploy-hook-token> is the last part of your deploy hook URL (rh_…). If you keep the whole hook URL in a secret, add /verdict to it:

curl -s "$METRICKLE_DEPLOY_HOOK/verdict?version=1.4.0"

Leave out version to get the latest release. version is the version you recorded the release with: the one you posted to the hook, the tag or short commit GitHub, Vercel or Netlify sent, or the app version. Like the hook URL, the token is the credential, so keep it in your CI secrets.

Response

A 200 with the release's verdict as JSON. A shortened example:

{
  "release": {
    "id": "rel_…",
    "version": "1.4.0",
    "at": 1791374400000,
    "url": "https://github.com/acme/web/releases/v1.4.0",
    "description": "Checkout fixes"
  },
  "verdict": "dropped",
  "sentence": "Release 1.4.0 dropped “Sign up”: 52% → 38%, hardest for Safari users.",
  "lostPerMonth": 4800,
  "impact": { "people": 112, "slowerMs": null, "hoursPerWeek": null },
  "comparison": { "mode": "time", "share": null },
  "settlesAt": 1792584000000,
  "evaluatedAt": 1791396000000,
  "report": "https://app.metrickle.com/apps/app_…/releases/rel_…",
  "checks": [
    {
      "kind": "task",
      "name": "Sign up",
      "status": "dropped",
      "before": 0.52,
      "after": 0.38,
      "delta": -0.14,
      "lostPerMonth": 4800,
      "impact": { "people": 112, "slowerMs": null, "hoursPerWeek": null },
      "groups": [
        { "who": "Safari users", "effect": "dropped", "before": 0.48, "now": 0.07 }
      ]
    },
    {
      "kind": "goal",
      "name": "Started a trial",
      "status": "clear",
      "before": 0.098,
      "after": 0.095,
      "delta": -0.003,
      "lostPerMonth": null,
      "impact": { "people": null, "slowerMs": null, "hoursPerWeek": null },
      "groups": []
    }
  ]
}

The response is the source of truth for the exact fields and their shapes. The main ones:

Rates are fractions from 0 to 1. A negative delta is a drop.

Wait for the verdict in GitHub Actions

A verdict needs real usage, so right after a deploy it's usually too_early. This step polls until the verdict is something else or a time limit passes, writes the sentence to the job summary, and fails the job on dropped when FAIL_ON_DROP is true.

Run it as a separate job after the deploy, so a long wait doesn't hold up anything else.

release-verdict:
  needs: deploy
  runs-on: ubuntu-latest
  timeout-minutes: 300
  steps:
    - name: Wait for the release verdict
      env:
        METRICKLE_DEPLOY_HOOK: ${{ secrets.METRICKLE_DEPLOY_HOOK }}
        VERSION: ${{ github.ref_name }}
        WAIT_MINUTES: "240"
        FAIL_ON_DROP: "true"
      run: |
        deadline=$(( $(date +%s) + WAIT_MINUTES * 60 ))
        verdict=too_early
        while :; do
          body=$(curl -sf "$METRICKLE_DEPLOY_HOOK/verdict?version=$VERSION" || echo '{}')
          verdict=$(echo "$body" | jq -r '.verdict // "too_early"')
          if [ "$verdict" != "too_early" ] || [ "$(date +%s)" -ge "$deadline" ]; then break; fi
          echo "Too early to tell. Checking again in 5 minutes."
          sleep 300
        done
        sentence=$(echo "$body" | jq -r '.sentence // "No verdict yet."')
        report=$(echo "$body" | jq -r '.report // empty')
        echo "### Release $VERSION: $sentence" >> "$GITHUB_STEP_SUMMARY"
        [ -n "$report" ] && echo "[Open the report]($report)" >> "$GITHUB_STEP_SUMMARY"
        echo "Verdict: $verdict"
        if [ "$verdict" = "dropped" ] && [ "$FAIL_ON_DROP" = "true" ]; then exit 1; fi

Things to adjust:

• The version. Send the same version you recorded the release with. github.ref_name suits a workflow that runs on a tag. For releases from Vercel or Netlify, which record the short commit, use ${GITHUB_SHA::7}.
• How long to wait. How soon a verdict arrives depends on your traffic and the limits on your goals and protected flows. A busy app may get one within the hour, a quiet one may take days. The report's Too early to tell estimate is a good guide. If the wait runs out while it's still too early to tell, the step passes and posts what the report says so far.
• What fails the job. watch (Early signal) never fails it here. Change the last line if you want it to.

To post the sentence to a pull request or Slack instead, pass $sentence and $report to the tool you use for that.

Hear about drops without CI

You don't need CI to hear about a drop. When a release's verdict becomes Drop found, Metrickle sends alert.release_dropped once, by email and to any Slack, Teams or webhook destination you've chosen it for. release.verdict carries a release's first verdict when it's No drop or Early signal.