The Latest Trends in App Testing for Mobile Games

You’ve been working on an app for months and it’s finally ready.  Now the big question – will it be a success?  The only way to find out is to put it in front of users.

Most developers get their games out to friends and family to get feedback, which is great for seeing usability flaws or where you may need to explain things better in your first time user experience (FTUE).

But nothing really compares to getting a large cohort of real random users to reflect the broader marketing so you can really see if what you’ve put together achieves the benchmark engagement and retention levels you were looking for.

Here are some tips around testing based on what I’ve seen and feedback from other developers (bear in mind this is Q1 2015 and things can change drastically):

What to Test?

There are two core things you want to test in this early stage: the on-boarding process and retention.  In the on-boarding process, are users actually understanding how to get through your initial process – whether that be registration or through your tutorial.  Ideally you want to streamline any pinch-points and get users engaged in your game.  Each point of your on-boarding process should be tracked so that you can tell for each install cohort what percentage of users get to the next step.  Hopefully you’ve captured some of the sticking points while watching users play your app, but having a scaled test will really point out where issues arise.

Retention is the next piece – what percent of the users that install on day one end up coming back the next day (often referred to as day 1 retention)?  If you are a successful smash like CrossyRoad (iOS | Android), you get phenomenal 65% day 1 retention; if you are below 40% then something is not really engaging users and getting them back to the game.  Equally important is looking at day 3 and day 7 retention – you might have a great first couple days, but users fall off dramatically.  Day 15 you’d want to see around 15% (all of us can’t attain the phenomenal 20% day 30 retention CrossyRoad has achieved)

CrossyRoad has achieved impressive 65% day 1 retention and 20% day 30 retention
CrossyRoad has achieved impressive 65% day 1 retention and 20% day 30 retention

Other things to test:

  • You are NOT going to get a good feel for monetization – for the most part you need a really big cohort of PAYING users to get statistically significant monetization stats.  If you see 2% conversion of your installed users that buy, that means to get 400-1000 BUYERS you’d need 20-50K installs in a day – and you’re generally not going to do that in your testing phase.  You can get directionally that it’s monetizing, but not something you can call predictive in any way.
  • You should test game and economy balancing, to see if it you’ve addressed any concerns or new ones crop up
  • If doing an ad buy to get your install cohorts, you can definitely test the creative you use so that you’ve got the best combinations by the time you do your global launch

What Platform?

One of the great things from the Game Developers Conference (GDC) in San Francisco is hearing developers war stories – what they found out and how they changed their games to improve these metrics.  What was consistent, though not necessarily surprising, was that every developer stressed that they tested on Android first, even if they ended up globally launching on iOS.  

Why do developers across the board iterate and perfect their game on Android instead of iOS?  Because it's faster to make iteractions.
Why do developers across the board iterate and perfect their game on Android instead of iOS? Because it’s faster to make iteractions.

Here’s why: You can iterate faster because the approval process can stretch over a week on Apple.   Luke Muscat of Halfbrick Games shared at GDC that they set up a process of iteration where they would publish the app in Google Play on Friday, get installs over the weekend, and see the stats on Monday morning.  Based on this they would make pretty significant iterations during the week, spend Friday testing it, and submitting it, then repeat the process.  This allowed them to compare similar day of the week traffic and iterate each week, more efficiently leveraging their development resources.  If you had to wait for over a week to get the game approved and in the marketplace, you are doubling the optimization time – that much further from deciding whether you can fix it or should focus your resources on the next project.

Almost every developer does their initial iterations on Android primarily because Apple reviews can take over a week.  On left, Apple's tool Testflight; at right HockeyApp that is commonly used across both platforms.
Almost every developer does their initial iterations on Android primarily because Apple reviews can take over a week. On left, Apple’s tool Testflight; at right HockeyApp that is commonly used across both platforms.

Apple specifically says they don’t want to have “beta” releases in their market – only finished products.  For testing they purchased and integrated Testflight into their development kit – every app submitted to iOS needs to have a Testflight integration.  But Testflight is only limited to users you have the email address for, which can definitely bias the audience selection you do for your test and limits the audience size (more on that below).

What Markets?

While traditionally developers have gone to Canada and Australia to test out their games, the escalating cost of installs has driven testing to the Philippines and Singapore
While traditionally developers have gone to Canada and Australia to test out their games, the escalating cost of installs has driven testing to the Philippines and Singapore

Keith Katz of Execution Labs mentioned at GDC that “Southeast Asia is the new Canada.”  Generally speaking, developers take the first versions of their apps and look for test markets similar to the most lucrative US and UK markets: Australia and Canada.  But over the last few years these markets have been inundated with developers buying a disproportionate percentage of the market and have increased the costs of acquisition.  Now the Philippines and Singapore, with largely English-speaking players – are more cost-effective options.

You still may want to benchmark one of these markets vs. the Australia and Canada markets, just so you can see if there are consistent differences in retention.

Another pro-tip that is done by some large developers is to release the game under a different name and under a different developer/publisher name.  This allows you to not get noticed by the press and eliminate the potential to be picked up by Editors to be featured on the platform when all you really want to do is see if the game is working as you expected

How Many Users?

Ideally you want a statistically significant sample of installs.  If you can get 1,000 users in an install cohort, you can have a 95% confidence of your result within + or -3%.  Your budget may not enable you to get that many users, so at the bare minimum I would have 400 users which would give you a 95% confidence in your results within + or – 5%.  This is the other limitation of Apple’s Testflight – you can only invite up to 1,000 users, so it’s very hard to get the necessary scale without going to a live release.

If you need to do multiple waves of cohorts to test your retention numbers, you are going to need more than the 1,000 email limit imposed by Apple's Testflight
If you need to do multiple waves of cohorts to test your retention numbers, you are going to need more than the 1,000 email limit imposed by Apple’s Testflight

Other Ways to Test?

Again this is a general guide to testing your apps from the mobile game industry perspective in Q1 of 2015 – things are bound to change as we see further and further consolidation in the game app market.  What else is working for you?  Always interested in hearing other perspectives on how developers get their games tweaked before launching globally, so drop me a line or comment below!

UPDATE

One of my followers noted that they heard South Africa (ZA) was a good and cheaper proxy as well.  I haven’t tested there, but thought would look at the potential reach in South Africa versus other markets.  Facebook continues to be one of the more cost-effective acquisition channels, so looking at their stats, here is the breakdown of potential reach by platform:

Android (version 4.0+)

  • US 82 Mil
  • UK 13.6 Mil
  • CA 6.6 Mil
  • AU 4.4 Mil
  • PH 20.0 Mil
  • SG 2.0 Mil
  • ZA 3.8 Mil

iOS (version 7.0+)

  • US 84 Mil
  • UK 17.8 Mil
  • CA 9.4 Mil
  • AU 7.2 Mil
  • PH 5.2 Mil
  • SG 1.5 Mil
  • ZA 0.8 Mil

Kicking Pay Walls Further Down the Road

evcinnyc_jolly_jam_match_mechanic
Jolly Jam innovates with a new matching mechanic, but the interesting thing is how much content is available to a new user

Rovio Stars’ new app Jolly Jam has been getting attention – and rightly so – for a new match three mechanic: instead of dragging or swapping tiles to match three objects or more in a row, users select a rectangle where the two corners must be the same character and everything within that rectangle that matches those characters is removed from the board. 

It’s great to see innovation in match 3 puzzles. But the other thing that I noticed when I played through it last week was how much game play I was able to run through before getting hit with a gate – I easily played over 30 levels over an hour and a half.  That’s a lot of play time right out of the gate.  And games like Best Fiends by Seriously (a studio by previous Rovio execs) had a similar very easy early on ramp.  So what’s going on?

Getting You Hooked

This is a general trend I’m seeing with a lot of casual free to play (F2P) games: we’re giving players access to more unencumbered content to get them hooked and engaged.  When a user spends an hour playing your game, they are making a hefty investment of time.

Maps like this one in Jolly Jam are now in 20% of the top games, reminding users the amount of time they've invested in the game
Maps like this one in Jolly Jam are now in 20% of the top games, reminding users the amount of time they’ve invested in the game

In conjunction with this, we’re seeing more casual puzzle game adopt a visual map of progress.  (The success of Candy Crush bred a lot of adotpion). Besides providing a light leaderboard showing where you are versus other players, it also provides a strong reminder of just how much time a player has invested in a game.

I haven’t combed through the top app charts, but according to the speakers at the Year in F2P Games at GDC this week, over 20% of mobile games in the top charts now employ some sort of map overview that shows the user’s progress (and more importantly investment or time) in the game.

Time Investment and Monetization

I think that it’s pretty clear that showing a player’s investment in the game can definitely help retention, but can it also help monetization?

One of the biggest money drivers in these puzzle games is when a player is just 2-3 moves short of completing a level, they are prompted to spend currency to get an additional pack of moves (echoing classic arcade games prompting you to put in another quarter to continue your game). 

Can the "pay another quarter to continue your game" mechanic be further impacted by the time invested in the round?
Can the “pay another quarter to continue your game” mechanic be further impacted by the time invested in the round?

I’d argue the time spent in playing a round (some of the later rounds in Candy Crush can take over 15 minutes) is also a psychological driver (do I want to spend another 15 minutes and try again?)  in getting players to fork over that extra quarter.  I don’t have data behind this, but it definitely bears testing.

Raising the Stakes (and the Value)

Bottom line, the top developers are creating a lot more content for players in order to get them hooked, getting users deeper into the game and hoping the investment of time ends up driving users to stay (and pay) longer.  This increases users expectations (and I’d argue in a good way) that upon downloading a free to play game, there is not just five minutes of play and a pay wall, but a deeper initial experience to enjoy.  Monetization only really begins to be a conversation (an exchange worth considering) after players have fully realized the value from the game.

Can Katy Perry Learn from Kim Kardashian?

While I showed in my last post how some developers have integrated ads well to make them a seamless and even valuable part of game play, Kim Kardashian: Hollywood (iOS | Android) is a great example where ads came in as an afterthought.

Kim Kardashian: Hollywood grosses an estimated $100K per day in IAP revenues on iOS
Kim Kardashian: Hollywood grosses an estimated $100K per day in IAP revenues on iOS

Regardless of what you think of the Kardashians (E! thinks they are worth $100 million over four years), Glu’s Kim Kardashian:Hollywood game is well crafted with detailed story arcs and a fine tuned energy mechanic. That drive not to fail your date, your fans or Kim’s wishes has kept the app solidly in the top 20 grossing games list and generating an estimated $100K a day from in-app purchases on iOS.

The ad integration though is a bit jarring: sporadically when you leave a location, the screen goes white and a static overlay ad is displayed.  It competes a bit with other things that happen when you leave a location, like calls from different characters, and feels out of place.

A more interesting and integrated approach would be to insert the ads when you travel from location to location.  Rather than take the bus from LAX to Hollywood, take an upgraded Uber for free (or less game cash) by watching a video ad.  What a better fit for this game than a brand picking up the tab.

Ads in Kim Kardashian: Hollywood could be more elegantly integrated into the game flow, having brands pick up the tab for a ride in exchange for a video ad
Advertising could be more elegantly integrated into the game flow, having brands pick up the tab for a ride in exchange for a video ad

It’s  small tweak, but it provides a better integrated experience by giving value to the player and a halo effect to the advertiser.

In reality I know that for most large studios, the ad group is a different corporate function and not at the table during the design stage, but maybe Glu can add it to their new game featuring Katy Perry.

You and I Can’t Bank on Being Kim Kardashian

When your app is grossing $100K a day on iOS you have the luxury of not having ads.  Most notably King.com initially had ads in Candy Crush Saga but when the game took off they could afford to remove them and focus more on the user experience.  Kim Kardashian: Hollywood could forgo ads at this point and focus on in-game sponsorships of products that could be introduced in new quests and story arcs.

But for most of us, ads can be and are an integral part of the business model and as such should be part of every game’s core design.

All About the Data Around Marketing, Optimizing and Monetizing Games and Mobile Apps