When I started this blog back in early 2009, I was musing about Amazon selling MP3s and nascent measurement of user sentiment via social media. Then in June of 2009 I finally started playing with data around Facebook games. This was all pre-Farmville and it was still the wild west, which meant on Facebook you could actually build a business (and attain user data) depending on how fast you could move. Zynga built its business on moving fast.
October of 2010, shortly after we released Cupcake Corner as a dessert-themed Cafe World knock-off while at OMGPOP, marked the end of Facebook as a platform for “social” or “viral” growth.
I remember it well because shortly after we released, we were at the FB HQ and heard Zuck announce big changes to the Facebook newsfeed algorithm, which lead down the long path to gating access to users. It impacted games, but it also impacted businesses who had invested in generating large audiences on the platform that, over time, Facebook would make you pay to actually reach.
And that’s the rub – in the early days you could engage users on the platform and bring them into your experiences outside of the platform; then you pretty much had to build a business WITHIN the platform, but users were relatively easy to get via virality; today Facebook wields all the levers to drive traffic to a business built within Facebook. Why invest in building something on Facebook when they can slowly squeeze your ability to grow and profit?
There is a little bit of the old “wild west” game development experience happening in Messenger games which are slowly being migrated into the main Facebook app with its own tab. The games are channeling some of that PvP magic with a hint of old-school virality, and Facebook is providing developers incentives (promotion aka cheap traffic) to build games there, but ultimately Facebook owns your users (and over time will extract more from you to reach your own users) and with more developers vying for eyeballs, you’ll have to pay to maintain the traffic you see during these early days.
Apple as a platform had had similar promise early on, leaving a window for developers to build a booming business. Here Apple exchanged the value of their platform for 30% of your profits, but it was a pretty huge audience. Eventually, once the app store platform got saturated, user acquisition got expensive and pretty much priced out smaller game developers.
Now with the introduction of Apple Arcade – at a ridiculously low price point of $5 a month – focusing on apps from a select few developers, they have relegated the rest of us to the morass of the over-saturated app store (where paying 30% for the privilege now seems overly exorbitant for the access what Apple provides).
On top of that, I’ve seen that Apple is jumping into the mobile acquisition ad market with its deep pockets to promote Apple Arcade and drive subscriptions. I can only guess this spending will drive up the cost of acquisition ads developers use to drive app installs, adding a bit more salt to the wounds. Analysts are predicting Apple will have 12 million paying subscribers by the end of 2020.
Early in the last decade, companies like Zynga, King and Super Cell were built by the reach of the Facebook and Apple platforms, but as the decade came to a close, the platforms have become the powerful gatekeepers and ultimately are moving to not only own the customer paying relationship (ad views on Facebook, direct payments on Apple), but the user experience as well.
You’ve been working on an app for months and it’s finally ready. Now the big question – will it be a success? The only way to find out is to put it in front of users.
Most developers get their games out to friends and family to get feedback, which is great for seeing usability flaws or where you may need to explain things better in your first time user experience (FTUE).
But nothing really compares to getting a large cohort of real random users to reflect the broader marketing so you can really see if what you’ve put together achieves the benchmark engagement and retention levels you were looking for.
Here are some tips around testing based on what I’ve seen and feedback from other developers (bear in mind this is Q1 2015 and things can change drastically):
What to Test?
There are two core things you want to test in this early stage: the on-boarding process and retention. In the on-boarding process, are users actually understanding how to get through your initial process – whether that be registration or through your tutorial. Ideally you want to streamline any pinch-points and get users engaged in your game. Each point of your on-boarding process should be tracked so that you can tell for each install cohort what percentage of users get to the next step. Hopefully you’ve captured some of the sticking points while watching users play your app, but having a scaled test will really point out where issues arise.
Retention is the next piece – what percent of the users that install on day one end up coming back the next day (often referred to as day 1 retention)? If you are a successful smash like CrossyRoad (iOS | Android), you get phenomenal 65% day 1 retention; if you are below 40% then something is not really engaging users and getting them back to the game. Equally important is looking at day 3 and day 7 retention – you might have a great first couple days, but users fall off dramatically. Day 15 you’d want to see around 15% (all of us can’t attain the phenomenal 20% day 30 retention CrossyRoad has achieved)
CrossyRoad has achieved impressive 65% day 1 retention and 20% day 30 retention
Other things to test:
You are NOT going to get a good feel for monetization – for the most part you need a really big cohort of PAYING users to get statistically significant monetization stats. If you see 2% conversion of your installed users that buy, that means to get 400-1000 BUYERS you’d need 20-50K installs in a day – and you’re generally not going to do that in your testing phase. You can get directionally that it’s monetizing, but not something you can call predictive in any way.
You should test game and economy balancing, to see if it you’ve addressed any concerns or new ones crop up
If doing an ad buy to get your install cohorts, you can definitely test the creative you use so that you’ve got the best combinations by the time you do your global launch
What Platform?
One of the great things from the Game Developers Conference (GDC) in San Francisco is hearing developers war stories – what they found out and how they changed their games to improve these metrics. What was consistent, though not necessarily surprising, was that every developer stressed that they tested on Android first, even if they ended up globally launching on iOS.
Why do developers across the board iterate and perfect their game on Android instead of iOS? Because it’s faster to make iteractions.
Here’s why: You can iterate faster because the approval process can stretch over a week on Apple. Luke Muscat of Halfbrick Games shared at GDC that they set up a process of iteration where they would publish the app in Google Play on Friday, get installs over the weekend, and see the stats on Monday morning. Based on this they would make pretty significant iterations during the week, spend Friday testing it, and submitting it, then repeat the process. This allowed them to compare similar day of the week traffic and iterate each week, more efficiently leveraging their development resources. If you had to wait for over a week to get the game approved and in the marketplace, you are doubling the optimization time – that much further from deciding whether you can fix it or should focus your resources on the next project.
Almost every developer does their initial iterations on Android primarily because Apple reviews can take over a week. On left, Apple’s tool Testflight; at right HockeyApp that is commonly used across both platforms.
Apple specifically says they don’t want to have “beta” releases in their market – only finished products. For testing they purchased and integrated Testflight into their development kit – every app submitted to iOS needs to have a Testflight integration. But Testflight is only limited to users you have the email address for, which can definitely bias the audience selection you do for your test and limits the audience size (more on that below).
What Markets?
While traditionally developers have gone to Canada and Australia to test out their games, the escalating cost of installs has driven testing to the Philippines and Singapore
Keith Katz of Execution Labs mentioned at GDC that “Southeast Asia is the new Canada.” Generally speaking, developers take the first versions of their apps and look for test markets similar to the most lucrative US and UK markets: Australia and Canada. But over the last few years these markets have been inundated with developers buying a disproportionate percentage of the market and have increased the costs of acquisition. Now the Philippines and Singapore, with largely English-speaking players – are more cost-effective options.
You still may want to benchmark one of these markets vs. the Australia and Canada markets, just so you can see if there are consistent differences in retention.
Another pro-tip that is done by some large developers is to release the game under a different name and under a different developer/publisher name. This allows you to not get noticed by the press and eliminate the potential to be picked up by Editors to be featured on the platform when all you really want to do is see if the game is working as you expected
How Many Users?
Ideally you want a statistically significant sample of installs. If you can get 1,000 users in an install cohort, you can have a 95% confidence of your result within + or -3%. Your budget may not enable you to get that many users, so at the bare minimum I would have 400 users which would give you a 95% confidence in your results within + or – 5%. This is the other limitation of Apple’s Testflight – you can only invite up to 1,000 users, so it’s very hard to get the necessary scale without going to a live release.
If you need to do multiple waves of cohorts to test your retention numbers, you are going to need more than the 1,000 email limit imposed by Apple’s Testflight
Other Ways to Test?
Again this is a general guide to testing your apps from the mobile game industry perspective in Q1 of 2015 – things are bound to change as we see further and further consolidation in the game app market. What else is working for you? Always interested in hearing other perspectives on how developers get their games tweaked before launching globally, so drop me a line or comment below!
UPDATE
One of my followers noted that they heard South Africa (ZA) was a good and cheaper proxy as well. I haven’t tested there, but thought would look at the potential reach in South Africa versus other markets. Facebook continues to be one of the more cost-effective acquisition channels, so looking at their stats, here is the breakdown of potential reach by platform:
