Tag Archives: Scalability

Mafia Wars Gets Whacked: The Woes of Revamping the Infrastructure of a Hit

Service issues, partially loaded screens and just not enough servers to manage a recently reconfigured game has caused Mafia Wars to take a nose dive in daily active users, dropping it back to levels not seen since August of this year:

zynga-mafia-wars-dau-drop-20091215

One could say it’s a perfect storm of issues (there was 48 hour of Facebook platform performance issues on the 9th and 10th that didn’t help last week’s roll out), but part of it I believe is an attempt to better improve the infrastructure. Back in early November, Zynga rolled out a performance enhancement on a Friday only to be forced to roll it back by two days later due to browser settings making it difficult for everyone to play the game.

Last Tuesday, the 8th, there were already signs that a new roll out of i-frame technology and anti-hacking measures were causing unintended issues. Somehow this code change was creating a cavalcade of issues, resulting in a product manager chat with users and an emergency bulletin board note to users on Saturday (the thread since removed), summarizing the issues and how they were intending to resolve them:

  • losing mafia/stats starting today – this is because our servers are being overwhelmed. We’re trying to add machines as fast as possible. We’ve stabilized the game for now and look to add more machines this coming week. Your mafias should all be back to normal now.
  • partial page loads – this is also related to the servers being overwhelmed. In the past, the app would have white screened, now it does partial loads. This issue will also be resolved when we add the servers next week.
  • lost items/progress – This was caused when we rolled out our performance upgrade. People who had the app open while we rolled out the upgrade had their accounts screwed up. In order to fix them, we had to revert them to whatever the most recent backup point was. We gave out items to anyone that we had to rollback to make up for the lost progress.

Major platform changes where you have millions of users online engaging with your game is not easy. No matter how good your QA, it’s darn near impossible to recreate a live scenario where as soon as you roll out, millions are hitting your servers. Equally daunting is finding a good time to have your game go off-line for maintenance, especially on a platform like Facebook where there are millions around the globe hitting your game.

For all intents and purposes, the Mafia Wars platform, launched over a year ago, was probably never developed for the sheer number of users it now supports – a number that has doubled in the last six months and would never have been predicted by even the most bullish of those in the social game space at the time Mafia Wars was developed. There are similar issues with Café World, the number two game on Facebook with nearly 10 million daily active users a day that has seen its numbers drop in the last week as well:

zynga-cafe-world-dau-drop-20091215

Users continue to complain of poor load times or the sim slowing down PC performance when left in the background. Again, this game was produced with some learnings from FarmVille, but the sim is much more animated than the typical farm.

With FarmVille, Zynga took steps to lessen the load on its servers, reducing animations and smooth effects (the introduction of chicken coops and dairy barns as well as storage is also a keen way to push more of the animated items off the screen). Now as Café World creaks under the load, there are small tweaks in place to simplify the animations (sparkling animation over finished dishes seems a bit less intense, pop-ups around dishes are less animated). It’s no wonder that they have yet to launch basic achievements in the game, fearing the impact of more users on a more robust game.

And that also puts in some interesting questions around other recent games like FishVille and PetVille – after fast growth, both games have tailed off a bit. Surely Zynga can advertise and drive more users, so are they holding off in part to ensure that the games can scale?

While Mafia Wars users may fret, you have to give props to Zynga for focusing on focusing resources explicitly to improve their infrastructure. The fallout from trying to fix Mafia Wars when it already is so hug only underscores the need to build your platform in such a way to make it easily extensible and with the option to add servers quickly on the fly when your estimates of traffic are completely blown away.

Three Fish Sim Games Swim Into Top 25

And then there were three. I had previously talked about the success of casual game Fish Tycoon being a precursor for the success of Tall Tree Games’ Fish World (currently #11 application on Facebook with 2.0 million daily active users), but two more fish/aquarium sim games have entered into the Top 25 on Facebook. My Aquarium by TwoFishes Interactive (#16 with 1.56 million DAU) and fast riser Happy Aquarium by CrowdStar (#17 with 1.48 million DAU) have both passed by Playfish’s latest game, Country Store.

Happy Aquarium is growing at a much faster pace than the other two, adding over 166,000 users a day over the last week, compared to around 72,000 a day for Fish World and My Fishbowl:

So why the fast growth?Could it be the persistent and somewhat pushy request at each login to continually spam your friends to join you? Well, all three do that – in fact the only difference apparently is the game name (see Fish World and Happy Aquarium at right).

Other “social spam” to brag or entice users’ friends to join in? Where Fish World and My Fishbowl has the “gift request” popular with Mafia Wars and FarmVille, Happy Aquarium only has the ability to give your friends fish food – which is critical to growing your fish – but less vibrant. And there is the standard ability to take pictures of your tank to post to your feed and the prompts at each leveling up to brag about it.

Regardless of the social spam opportunities to push the game, users only will forward it to their friends if they love the game. Ratings for the games nearly match their current growth rates: Happy Aquarium has 4.1 stars out of five, Fish World with 3.6 and My Fishbowl at 3.4. Happy Aquarium is easier to clean, requiring only a click of the mouse, where Fish World requires you to move your mouse back and forth and actually scrub the tank. There also may be some regional biases going on here which I’d have to look at further, as My Fishbowl seems to skew towards users in Asia, just looking at the number of reviews on the site, which means it may not have penetrated the US and European Facebook audiences yet.

Growth is Hard – Flushing the Fish

Fish World’s faster than anticipated growth required it to add more servers and in doing the upgrade had a hardware failure resulting in data loss for several players (THAT’s why my tank is full of dead fish, and NOT that I didn’t feed them yesterday). Coins and “Fish Bucks” were given to those impacted, but judging from the response of users, it looks like not enough to replace what they lost and many are moving on. Will be interesting how many flush Fish World down the proverbial Facebook Application toilet bowl. Lesson to developers – don’t roll out your game without a quick and painless way to upgrade your servers and have a contingency communication plan set up if you run into problems.