You’ve been working on an app for months and it’s finally ready.  Now the big question – will it be a success?  The only way to find out is to put it in front of users.

Most developers get their games out to friends and family to get feedback, which is great for seeing usability flaws or where you may need to explain things better in your first time user experience (FTUE).

But nothing really compares to getting a large cohort of real random users to reflect the broader marketing so you can really see if what you’ve put together achieves the benchmark engagement and retention levels you were looking for.

Here are some tips around testing based on what I’ve seen and feedback from other developers (bear in mind this is Q1 2015 and things can change drastically):

What to Test?

There are two core things you want to test in this early stage: the on-boarding process and retention.  In the on-boarding process, are users actually understanding how to get through your initial process – whether that be registration or through your tutorial.  Ideally you want to streamline any pinch-points and get users engaged in your game.  Each point of your on-boarding process should be tracked so that you can tell for each install cohort what percentage of users get to the next step.  Hopefully you’ve captured some of the sticking points while watching users play your app, but having a scaled test will really point out where issues arise.

Retention is the next piece – what percent of the users that install on day one end up coming back the next day (often referred to as day 1 retention)?  If you are a successful smash like CrossyRoad (iOS | Android), you get phenomenal 65% day 1 retention; if you are below 40% then something is not really engaging users and getting them back to the game.  Equally important is looking at day 3 and day 7 retention – you might have a great first couple days, but users fall off dramatically.  Day 15 you’d want to see around 15% (all of us can’t attain the phenomenal 20% day 30 retention CrossyRoad has achieved)

Other things to test:

• You are NOT going to get a good feel for monetization – for the most part you need a really big cohort of PAYING users to get statistically significant monetization stats.  If you see 2% conversion of your installed users that buy, that means to get 400-1000 BUYERS you’d need 20-50K installs in a day – and you’re generally not going to do that in your testing phase.  You can get directionally that it’s monetizing, but not something you can call predictive in any way.
• You should test game and economy balancing, to see if it you’ve addressed any concerns or new ones crop up
• If doing an ad buy to get your install cohorts, you can definitely test the creative you use so that you’ve got the best combinations by the time you do your global launch

What Platform?

One of the great things from the Game Developers Conference (GDC) in San Francisco is hearing developers war stories – what they found out and how they changed their games to improve these metrics.  What was consistent, though not necessarily surprising, was that every developer stressed that they tested on Android first, even if they ended up globally launching on iOS.  

Here’s why: You can iterate faster because the approval process can stretch over a week on Apple.   Luke Muscat of Halfbrick Games shared at GDC that they set up a process of iteration where they would publish the app in Google Play on Friday, get installs over the weekend, and see the stats on Monday morning.  Based on this they would make pretty significant iterations during the week, spend Friday testing it, and submitting it, then repeat the process.  This allowed them to compare similar day of the week traffic and iterate each week, more efficiently leveraging their development resources.  If you had to wait for over a week to get the game approved and in the marketplace, you are doubling the optimization time – that much further from deciding whether you can fix it or should focus your resources on the next project.

Apple specifically says they don’t want to have “beta” releases in their market – only finished products.  For testing they purchased and integrated Testflight into their development kit – every app submitted to iOS needs to have a Testflight integration.  But Testflight is only limited to users you have the email address for, which can definitely bias the audience selection you do for your test and limits the audience size (more on that below).

What Markets?

Keith Katz of Execution Labs mentioned at GDC that “Southeast Asia is the new Canada.”  Generally speaking, developers take the first versions of their apps and look for test markets similar to the most lucrative US and UK markets: Australia and Canada.  But over the last few years these markets have been inundated with developers buying a disproportionate percentage of the market and have increased the costs of acquisition.  Now the Philippines and Singapore, with largely English-speaking players – are more cost-effective options.

You still may want to benchmark one of these markets vs. the Australia and Canada markets, just so you can see if there are consistent differences in retention.

Another pro-tip that is done by some large developers is to release the game under a different name and under a different developer/publisher name.  This allows you to not get noticed by the press and eliminate the potential to be picked up by Editors to be featured on the platform when all you really want to do is see if the game is working as you expected

How Many Users?

Ideally you want a statistically significant sample of installs.  If you can get 1,000 users in an install cohort, you can have a 95% confidence of your result within + or -3%.  Your budget may not enable you to get that many users, so at the bare minimum I would have 400 users which would give you a 95% confidence in your results within + or – 5%.  This is the other limitation of Apple’s Testflight – you can only invite up to 1,000 users, so it’s very hard to get the necessary scale without going to a live release.

Other Ways to Test?

Again this is a general guide to testing your apps from the mobile game industry perspective in Q1 of 2015 – things are bound to change as we see further and further consolidation in the game app market.  What else is working for you?  Always interested in hearing other perspectives on how developers get their games tweaked before launching globally, so drop me a line or comment below!

UPDATE

One of my followers noted that they heard South Africa (ZA) was a good and cheaper proxy as well.  I haven’t tested there, but thought would look at the potential reach in South Africa versus other markets.  Facebook continues to be one of the more cost-effective acquisition channels, so looking at their stats, here is the breakdown of potential reach by platform:

Android (version 4.0+)

• US 82 Mil
• UK 13.6 Mil
• CA 6.6 Mil
• AU 4.4 Mil
• PH 20.0 Mil
• SG 2.0 Mil
• ZA 3.8 Mil

iOS (version 7.0+)

• US 84 Mil
• UK 17.8 Mil
• CA 9.4 Mil
• AU 7.2 Mil
• PH 5.2 Mil
• SG 1.5 Mil
• ZA 0.8 Mil