Soft-launch week two: turning the 72-hour crash list into a stable release

A ranked crash list leading to a rollout that widens step by step once each fix is confirmed

By the end of the third day of a soft launch, the team has a crash list and, often, little sense yet of which entries matter. Part one, our guide to triaging soft-launch crashes over the first 72 hours, covers how that list gets built. This piece picks up from there: how the list becomes a fix plan, how each fix gets checked, and how the store data tells you whether the next build is ready for a wider audience.

Week two is quieter than the first evening, and that is the risk. The urgent fires are out, the team wants to get back to features, and the rollout percentage is one click away from going up.

From a raw list to a fix plan

The 72-hour list is usually a mix of real defects, duplicates under different names, and one-off reports nobody could repeat. Before anyone opens a code editor, turn it into a plan the whole team can read:

  • Merge entries that share a stack signature, even if testers and players described them differently.
  • For each remaining entry, write down how many users it reached, which build versions show it, and which device models carry most of it.
  • Give every entry a named owner and one of three outcomes: fix in the next build, fix in a later build, or leave open and watch.
  • Note what proof will close it, for example a clean regression run on the device that reproduced it plus a week of store data on the new version.

A plan in this shape lets a producer see at a glance what the next build is meant to change, and lets QA plan the checks for it in advance.

Rank by people affected, not by event count

Raw event totals mislead. One phone stuck in a crash loop can log hundreds of events, while a crash that hits thousands of players once each looks small on the same chart. Rank by the number of users affected first, then by how badly it hurts them: lost progress, a broken purchase or a game that will not start come ahead of a crash on the way out of a menu.

Then look at devices. Compare each model's share of crashes with its share of your installs. A model that carries a tenth of your crashes but only a fiftieth of your players is a device problem worth its own ticket, and often its own fix.

On Android, Google's Android vitals documentation lists bad behaviour thresholds for the user-perceived crash rate of 1.09% overall and 8% for any single phone model, and 0.47% overall and 8% per phone model for the user-perceived ANR rate. Play uses the last 28 days of data to assess quality, so a fix shipped today takes time to show in the overall figure. Treat those thresholds as an outer limit for the plan, not as the goal.

On iOS, keep in mind what the numbers represent. Apple's guide to acquiring crash reports and diagnostic logs explains that the Crashes organizer in Xcode shows reports from customers who share diagnostic and usage information, while TestFlight users share crash reports automatically. It is a sample, so trends between builds tell you more than any single total. Upload symbol information with each build, or the reports will be hard to read when you need them most.

A regression pass for every fix build

Each fix build needs its own short pass before it goes anywhere near more players. Three parts are usually enough:

  • Confirm each fix on the device and OS version that reproduced the original problem, using the steps recorded in week one.
  • Rerun the main player flows that touch the changed code, for example the first launch, a save and reload, a purchase, and a session resumed after a call or a lock screen.
  • Check the fix on at least one device from a different family, because a change made for one chipset or OS version can move the problem elsewhere.

Keep the repro steps from every closed crash as permanent regression cases. A crash closed in one build can return in a later one, and those steps are the cheapest way to catch it. Record the exact build number on every result, so nobody signs off a build that was never tested.

Let the data confirm the fix before the rollout widens

A fix is only proven once the new version has been in real players' hands long enough to compare with the old one. Both stores give you a way to control that.

How the store controls themselves behave, and why old and new builds overlap while they run, is covered in our article on mixed-version QA for game updates. For week two, the useful habit is to treat each rollout step as a checkpoint with a written condition, rather than a timer that moves on by itself.

  • Google Play: the percentage in a staged rollout stays where you set it until someone raises it. Write down the crash and ANR figures the new version must show before each increase, and who makes that call. Countries cannot be removed once the rollout starts, so choose the first markets carefully.
  • App Store: a phased release moves up a step each day over a week unless you pause it. Check the new build in the Crashes organizer each morning, and pause if a crash you marked as fixed shows up again.

Whichever store, filter crash data to the new version only and compare it with the previous version over a similar number of sessions. The first day or two of a rollout carry small numbers, so a quiet start is encouraging but not yet proof.

When is a build stable enough to scale?

Agree the answer before the build is ready, so the decision is not made under pressure on the day. A useful set of conditions looks like this:

  • No open crash in the paths that cost players progress or money: starting the game, saving, purchases and resuming a session.
  • Crash and ANR rates for the new version below the store thresholds overall and on each of your most-installed device models.
  • Rates holding steady or falling as the rollout percentage rises, not only at the smallest step.
  • The regression pass on that exact build complete, with any failures fixed or written down as known issues.
  • A short list of what is still open, who owns each item and which later build should close it.

When those hold, scaling is a business call with evidence behind it. When one of them does not, the useful question is whether the gap is small enough to accept for this market, and that call belongs to the producer with the data in front of them.

Planning the build after soft launch?

GameCloud has tested games for studios since FY 2010-11, and our testers cover PC, iOS, Android, TV, Web, XR and other emerging platforms. When your soft-launch crash list needs turning into regression passes and a release you can widen with confidence, send the build, the platforms in scope and your current crash summary to Sales@GameCloud-Ltd.com, and we can talk through how we would cover week two.