Release-Over-Release Error Impact: Did This Deploy Break It?

Your error count spiked at 3 p.m. The deploy shipped at 2:58 p.m. Correlation is causation, right? Not quite. Traffic doubled when a news outlet linked to you. Your marketing team launched a campaign. A third-party API went down and every user hit the retry loop. A bot wave hammered your endpoint. Release-over-release error impact is the discipline of answering "did this deploy actually break it?" by comparing the right metrics between releases, not just watching raw counts climb. Without that rigor, you'll roll back good deploys and miss real regressions.

This guide teaches you how to set up the comparison, what metrics to use, and the adoption trap that invalidates almost every release comparison teams run.

Rates, not counts

The foundational mistake is comparing raw error counts. If release A had 100 errors and release B had 200 errors, release B looks twice as bad. But if release B had twice as much traffic—double the sessions, double the API calls, double the user load—then it's actually the same quality, just scaled up.

Always compare rates, not counts.

The simplest rate is error rate per session: divide total errors by total sessions. If release A logged 100 errors across 10,000 sessions (1% error rate) and release B logged 200 errors across 20,000 sessions (also 1% error rate), they're equivalent in quality. Release B didn't break anything; it just served more users.

error_rate_per_session = total_errors / total_sessions

Similarly, compare crash-free rate release-over-release. A 99.0% crash-free rate on release A vs 98.5% on release B tells you something real about stability. A raw count of 500 crashes on one release vs 1,200 on another only tells you which release got more traffic.

The reason this matters: every deploy increases traffic slightly (new release hype, auto-updates, canary expansion). If you don't normalize, you'll be chasing phantom regressions created by growth, not actual bugs. Normalize first, and you'll see the signal clearly.

The deltas worth tracking

Once you're comparing rates, focus on these four signals. Each one tells you a different story about what a deploy actually changed.

New issues

New issues are errors that first appeared in the new release—their first-seen version matches the release you're investigating. These are the clearest deploy signal. If your release ships and ten brand-new stack traces appear, something in your code changed to trigger them.

The catch: new issues can be false positives. A third-party library that was already broken might not manifest until a code path changes in your release and hits it. Or a race condition might get exposed by different request timing. Still, a spike in new-issue count is the strongest "this deploy broke something" indicator.

Regressions

A regression is an issue that was previously resolved (fixed in an older release) and reappeared in the new one. LightTrace automatically reopens a resolved issue if the same stack trace reappears after the version that fixed it. If release A fixed bug X, and bug X reappears in release C, the issue moves from "resolved" back to "regressed in C."

Regressions are rare but catastrophic. They're usually caused by a revert, a conflicting fix, or a cherry-pick gone wrong. They're also often the fastest to understand: you can look at the commit that resolved it before and ask whether this release undid it.

Resolved issues

Issues that stopped in your new release confirm that a fix shipped. If you're monitoring for regression, also celebrate the inverse: errors that used to spike every day but vanished after your deploy. These offset new issues and show the positive impact of your work.

Crash-free rate delta

Compare crash-free rate release-over-release. A 1–2% drop hour-over-hour is a regression signal. A 0.5% drop might be noise; a 2% drop is usually a real problem. The threshold depends on your app and users, but the direction is always meaningful. Track it closely in the first hours after deployment.

Error rate per session

The headline number: total errors divided by sessions. If this metric jumps 50% or more, something changed. If it stays flat or improves, your deploy was stable even if raw counts went up.

Adoption skew — the trap that fools everyone

Here's the trap that invalidates most release comparisons: adoption is never uniform.

When you ship a new release, the first people to get it are early updaters. They're often on newer devices, better networks, more predictable hardware (corporate laptops), and more forgiving environments. They have fewer background apps running. Their data plans are faster. They're more likely to click "update now" instead of "remind me later."

Compare a release at 5% adoption to one at 95% adoption, and you're comparing an elite group to the general population. The 5%-adoption release might look stable simply because you're seeing only power users and haven't hit edge cases yet. Cable connections. Overloaded devices. Users in tunnels or on trains. The Android 8 phone with 1 GB of RAM. Wait a week, and adoption climbs, and suddenly the crash-free rate drops 3% because you're now hitting real-world conditions.

The rule: compare only at comparable adoption. If release A is at 90% adoption and release B is at 5%, you don't have a fair comparison yet. Either give release B time to soak and reach 90%, or accept that you're comparing different populations and adjust expectations.

For services and web apps, this isn't as acute (your users auto-refresh), but for mobile apps, adoption skew can hide regressions for hours or days. Know this trap, and you'll stop making panic rollback decisions based on phantom differences.

The comparison window

Once adoption is comparable, pick equal-length time windows. Don't compare "since the deploy" (which varies based on when issues hit) to "last 7 days" (which is a fixed window). Compare the first 24 hours of release A to the first 24 hours of release B. Or the first 7 days to the first 7 days.

Avoid the first-hours skew. The first 2–4 hours after a deploy often show elevated error rates due to cache warm-up, connection pool initialization, and migrations. Many teams see a small spike, then a return to baseline. Don't judge a release on its first hour; wait for 4–6 hours of data.

Also, compare against the same day-of-week if possible. Deploy on a Wednesday? Compare to the previous Wednesday, not Monday. Traffic patterns differ by day, and you want to control for that.

Attribution beyond the version

Knowing that a release regressed is step one. Knowing why is step two. LightTrace helps here by:

• Tagging every error with its first-seen release, so you know which deploy introduced the issue.
• Offering GitHub source links — click a stack frame and jump to the exact line that threw the error. From there, look at the diff since the previous release.
• Commit range — LightTrace can show you the exact commits shipped in this release vs the last. Correlate the error stack trace with changes to that code path.

Link GitHub source links with error grouping and release tagging, and you can move from "this release is unstable" to "this specific line in this specific commit is throwing" in seconds.

What to do with the answer

Once you've confirmed a regression, decide before the next deploy whether you'll roll back or fix forward.

Roll back is the safe move: revert the release, deploy the previous version, and fix the issue on a branch. This is fastest for users and simplest operationally. You lose a few hours of work but you gain certainty.

Fix forward means keeping the broken release live while you fix the bug and deploy again. This is faster if the fix is simple (typo, missing null check), but riskier if the issue affects many users. Set a threshold before deploying: if the crash-free rate drops more than 2%, you roll back. If it's 0.5%, you fix forward. Know your SLO and your risk tolerance.

Deployment rollback strategies and canary deployments go deeper on both approaches. The key is deciding upfront, not in a panic at 4 a.m.

Putting it together: a comparison checklist

Start with release health monitoring to tag errors with their release. Use LightTrace's crash-free rate dashboard to set baselines. Then run this comparison every deploy: are the rates better, the same, or worse than last time? Did adoption reach parity? Did adoption skew fool us? Set alert thresholds before you ship, so you know when to roll back.

Most false alarms come from comparing incomparable releases (adoption mismatch, time-of-day bias) or watching raw counts (traffic growth, not regression). Fix those two mistakes, and you'll catch real regressions and avoid panic rollbacks.

The discipline of comparing releases saves you hours of debugging and dozens of unnecessary rollbacks. Set it up once in your CI/CD, and every future deploy gets instant visibility into whether it actually broke anything.