Mobile Games, Virtual Reality

The Reality of VR: Game Devs Speak Up at GDC

Leave a comment

One of the things I tried to get my arms around at GDC this year was what exactly is the opportunity with VR – as the mobile game industry begins to consolidate, could VR be the next big wave?

Matthew Falcus posted a nice overview in Gamastura today — Making the Most of the VR Opportunity — that provides a great overview.  I wanted to add some of my notes from GDC which I thought were interesting context as well, especially from day one of the Virtual and Augmented Reality Roundtable session.

Indie VR Developers by the Numbers

  • The room was to capacity with about 80 participants
  • 8 people had shipped a game, 60% of the participants were actively developing something
  • 2 had been working since the late 90s in Augmented Reality/Virtual Reality – seeing a lot of the same issues that people were tackling then now resurfacing today
  • 95% of the participants were male
  • About 95% were developing in Unity because it’s easy and can integrate things right away.  One or two were using proprietary code, mostly because they started a while ago and just stuck with it

You Are Going to Get Sick

A lot of folks were sharing their experiences – for example you are going to make yourself sick when you are prototyping and can’t work the rest of the day – but the bottom line was that every user is different and there is no magic bullet to solve sickness – yet.  That said here are some of the things that participants mentioned seemed to make it better for users:

  • Made the game explicitly have the character put on goggles
  • Having a focal point in front of the user helps keep their balance
  • Make steps/staircase feel like a ramp instead of bouncing up and down them
  • Team Fortress 2 developers ended up creating a “meathook” solution where it’s all about your head orientation – your feet are just dangling
  • Bring content to the user instead of user moving to the content (Oculus Rift’s Crescent Bay demo did this)
  • Teleport to locations vs. the manual walking, moving

If you are developing a game then, how do you get around if allowing a user to walk around and explore the environment makes some people sick?  It’s a different narrative and thought process.

Is the Tech There Yet?

Generally, the answer is sort of.  There were several new devices debuted that week and discussions about trying to develop to the lowest common form (John Carmack of Oculus suggested developers look at graphics like they were building for the Game Cube) because no one knows which platform will take off.

This discussion veered into issues about the lack of tactile feedback and the disconnect between not being able to use your hands or see yourself within VR.  One participant said the current state of AR/VR now is analogous to what 3D was in the 80s, “We are at a point of finding the cheats for AR and VR and then systems will do the math to help us achieve the solution.”

The biggest disconnect for many though, is that it feels like the technology is driving the discussion, where instead it should be the experience we are trying to achieve in AR/VR should be driving the technology.  As one participant reacting to the discussion about the need for haptic feedback suits noted, “how long do you expect them to wear it?  An hour is probably a LOT for mainstream users.”

The Reality: VR Might Not be About Games at All

I’ve tried both the simple (Google Cardboard) and latest  (Oculus Crescent Bay) VR headsets and the experiences are intriguing, but not something I’m yearning to go back to over and over again.  One of the most interesting debates really is what IS the experience that is going to be the defining one for VR?   As Holden Link from Turbo Button noted during the VR for Indies Panel, games might not end up being the killer app.  “The killer app for VR is probably Netflix – being in a giant theater.”

 

Mobile Games

The Latest Trends in App Testing for Mobile Games

Leave a comment

You’ve been working on an app for months and it’s finally ready.  Now the big question – will it be a success?  The only way to find out is to put it in front of users.

Most developers get their games out to friends and family to get feedback, which is great for seeing usability flaws or where you may need to explain things better in your first time user experience (FTUE).

But nothing really compares to getting a large cohort of real random users to reflect the broader marketing so you can really see if what you’ve put together achieves the benchmark engagement and retention levels you were looking for.

Here are some tips around testing based on what I’ve seen and feedback from other developers (bear in mind this is Q1 2015 and things can change drastically):

What to Test?

There are two core things you want to test in this early stage: the on-boarding process and retention.  In the on-boarding process, are users actually understanding how to get through your initial process – whether that be registration or through your tutorial.  Ideally you want to streamline any pinch-points and get users engaged in your game.  Each point of your on-boarding process should be tracked so that you can tell for each install cohort what percentage of users get to the next step.  Hopefully you’ve captured some of the sticking points while watching users play your app, but having a scaled test will really point out where issues arise.

Retention is the next piece – what percent of the users that install on day one end up coming back the next day (often referred to as day 1 retention)?  If you are a successful smash like CrossyRoad (iOS | Android), you get phenomenal 65% day 1 retention; if you are below 40% then something is not really engaging users and getting them back to the game.  Equally important is looking at day 3 and day 7 retention – you might have a great first couple days, but users fall off dramatically.  Day 15 you’d want to see around 15% (all of us can’t attain the phenomenal 20% day 30 retention CrossyRoad has achieved)

CrossyRoad has achieved impressive 65% day 1 retention and 20% day 30 retention
CrossyRoad has achieved impressive 65% day 1 retention and 20% day 30 retention

Other things to test:

  • You are NOT going to get a good feel for monetization – for the most part you need a really big cohort of PAYING users to get statistically significant monetization stats.  If you see 2% conversion of your installed users that buy, that means to get 400-1000 BUYERS you’d need 20-50K installs in a day – and you’re generally not going to do that in your testing phase.  You can get directionally that it’s monetizing, but not something you can call predictive in any way.
  • You should test game and economy balancing, to see if it you’ve addressed any concerns or new ones crop up
  • If doing an ad buy to get your install cohorts, you can definitely test the creative you use so that you’ve got the best combinations by the time you do your global launch

What Platform?

One of the great things from the Game Developers Conference (GDC) in San Francisco is hearing developers war stories – what they found out and how they changed their games to improve these metrics.  What was consistent, though not necessarily surprising, was that every developer stressed that they tested on Android first, even if they ended up globally launching on iOS.  

Why do developers across the board iterate and perfect their game on Android instead of iOS? Because it's faster to make iteractions.
Why do developers across the board iterate and perfect their game on Android instead of iOS? Because it’s faster to make iteractions.

Here’s why: You can iterate faster because the approval process can stretch over a week on Apple.   Luke Muscat of Halfbrick Games shared at GDC that they set up a process of iteration where they would publish the app in Google Play on Friday, get installs over the weekend, and see the stats on Monday morning.  Based on this they would make pretty significant iterations during the week, spend Friday testing it, and submitting it, then repeat the process.  This allowed them to compare similar day of the week traffic and iterate each week, more efficiently leveraging their development resources.  If you had to wait for over a week to get the game approved and in the marketplace, you are doubling the optimization time – that much further from deciding whether you can fix it or should focus your resources on the next project.

Almost every developer does their initial iterations on Android primarily because Apple reviews can take over a week. On left, Apple's tool Testflight; at right HockeyApp that is commonly used across both platforms.
Almost every developer does their initial iterations on Android primarily because Apple reviews can take over a week. On left, Apple’s tool Testflight; at right HockeyApp that is commonly used across both platforms.

Apple specifically says they don’t want to have “beta” releases in their market – only finished products.  For testing they purchased and integrated Testflight into their development kit – every app submitted to iOS needs to have a Testflight integration.  But Testflight is only limited to users you have the email address for, which can definitely bias the audience selection you do for your test and limits the audience size (more on that below).

What Markets?

While traditionally developers have gone to Canada and Australia to test out their games, the escalating cost of installs has driven testing to the Philippines and Singapore
While traditionally developers have gone to Canada and Australia to test out their games, the escalating cost of installs has driven testing to the Philippines and Singapore

Keith Katz of Execution Labs mentioned at GDC that “Southeast Asia is the new Canada.”  Generally speaking, developers take the first versions of their apps and look for test markets similar to the most lucrative US and UK markets: Australia and Canada.  But over the last few years these markets have been inundated with developers buying a disproportionate percentage of the market and have increased the costs of acquisition.  Now the Philippines and Singapore, with largely English-speaking players – are more cost-effective options.

You still may want to benchmark one of these markets vs. the Australia and Canada markets, just so you can see if there are consistent differences in retention.

Another pro-tip that is done by some large developers is to release the game under a different name and under a different developer/publisher name.  This allows you to not get noticed by the press and eliminate the potential to be picked up by Editors to be featured on the platform when all you really want to do is see if the game is working as you expected

How Many Users?

Ideally you want a statistically significant sample of installs.  If you can get 1,000 users in an install cohort, you can have a 95% confidence of your result within + or -3%.  Your budget may not enable you to get that many users, so at the bare minimum I would have 400 users which would give you a 95% confidence in your results within + or – 5%.  This is the other limitation of Apple’s Testflight – you can only invite up to 1,000 users, so it’s very hard to get the necessary scale without going to a live release.

If you need to do multiple waves of cohorts to test your retention numbers, you are going to need more than the 1,000 email limit imposed by Apple's Testflight
If you need to do multiple waves of cohorts to test your retention numbers, you are going to need more than the 1,000 email limit imposed by Apple’s Testflight

Other Ways to Test?

Again this is a general guide to testing your apps from the mobile game industry perspective in Q1 of 2015 – things are bound to change as we see further and further consolidation in the game app market.  What else is working for you?  Always interested in hearing other perspectives on how developers get their games tweaked before launching globally, so drop me a line or comment below!

UPDATE

One of my followers noted that they heard South Africa (ZA) was a good and cheaper proxy as well.  I haven’t tested there, but thought would look at the potential reach in South Africa versus other markets.  Facebook continues to be one of the more cost-effective acquisition channels, so looking at their stats, here is the breakdown of potential reach by platform:

Android (version 4.0+)

  • US 82 Mil
  • UK 13.6 Mil
  • CA 6.6 Mil
  • AU 4.4 Mil
  • PH 20.0 Mil
  • SG 2.0 Mil
  • ZA 3.8 Mil

iOS (version 7.0+)

  • US 84 Mil
  • UK 17.8 Mil
  • CA 9.4 Mil
  • AU 7.2 Mil
  • PH 5.2 Mil
  • SG 1.5 Mil
  • ZA 0.8 Mil
Casual Games, Mobile Games

Kicking Pay Walls Further Down the Road

1 Comment
evcinnyc_jolly_jam_match_mechanic
Jolly Jam innovates with a new matching mechanic, but the interesting thing is how much content is available to a new user

Rovio Stars’ new app Jolly Jam has been getting attention – and rightly so – for a new match three mechanic: instead of dragging or swapping tiles to match three objects or more in a row, users select a rectangle where the two corners must be the same character and everything within that rectangle that matches those characters is removed from the board. 

It’s great to see innovation in match 3 puzzles. But the other thing that I noticed when I played through it last week was how much game play I was able to run through before getting hit with a gate – I easily played over 30 levels over an hour and a half.  That’s a lot of play time right out of the gate.  And games like Best Fiends by Seriously (a studio by previous Rovio execs) had a similar very easy early on ramp.  So what’s going on?

Getting You Hooked

This is a general trend I’m seeing with a lot of casual free to play (F2P) games: we’re giving players access to more unencumbered content to get them hooked and engaged.  When a user spends an hour playing your game, they are making a hefty investment of time.

Maps like this one in Jolly Jam are now in 20% of the top games, reminding users the amount of time they've invested in the game
Maps like this one in Jolly Jam are now in 20% of the top games, reminding users the amount of time they’ve invested in the game

In conjunction with this, we’re seeing more casual puzzle game adopt a visual map of progress.  (The success of Candy Crush bred a lot of adotpion). Besides providing a light leaderboard showing where you are versus other players, it also provides a strong reminder of just how much time a player has invested in a game.

I haven’t combed through the top app charts, but according to the speakers at the Year in F2P Games at GDC this week, over 20% of mobile games in the top charts now employ some sort of map overview that shows the user’s progress (and more importantly investment or time) in the game.

Time Investment and Monetization

I think that it’s pretty clear that showing a player’s investment in the game can definitely help retention, but can it also help monetization?

One of the biggest money drivers in these puzzle games is when a player is just 2-3 moves short of completing a level, they are prompted to spend currency to get an additional pack of moves (echoing classic arcade games prompting you to put in another quarter to continue your game). 

Can the "pay another quarter to continue your game" mechanic be further impacted by the time invested in the round?
Can the “pay another quarter to continue your game” mechanic be further impacted by the time invested in the round?

I’d argue the time spent in playing a round (some of the later rounds in Candy Crush can take over 15 minutes) is also a psychological driver (do I want to spend another 15 minutes and try again?)  in getting players to fork over that extra quarter.  I don’t have data behind this, but it definitely bears testing.

Raising the Stakes (and the Value)

Bottom line, the top developers are creating a lot more content for players in order to get them hooked, getting users deeper into the game and hoping the investment of time ends up driving users to stay (and pay) longer.  This increases users expectations (and I’d argue in a good way) that upon downloading a free to play game, there is not just five minutes of play and a pay wall, but a deeper initial experience to enjoy.  Monetization only really begins to be a conversation (an exchange worth considering) after players have fully realized the value from the game.

All About the Data Around Marketing, Optimizing and Monetizing Games and Mobile Apps