Mafia Wars Gets Whacked: The Woes of Revamping the Infrastructure of a Hit

Service issues, partially loaded screens and just not enough servers to manage a recently reconfigured game has caused Mafia Wars to take a nose dive in daily active users, dropping it back to levels not seen since August of this year:

zynga-mafia-wars-dau-drop-20091215

One could say it’s a perfect storm of issues (there was 48 hour of Facebook platform performance issues on the 9th and 10th that didn’t help last week’s roll out), but part of it I believe is an attempt to better improve the infrastructure. Back in early November, Zynga rolled out a performance enhancement on a Friday only to be forced to roll it back by two days later due to browser settings making it difficult for everyone to play the game.

Last Tuesday, the 8th, there were already signs that a new roll out of i-frame technology and anti-hacking measures were causing unintended issues. Somehow this code change was creating a cavalcade of issues, resulting in a product manager chat with users and an emergency bulletin board note to users on Saturday (the thread since removed), summarizing the issues and how they were intending to resolve them:

  • losing mafia/stats starting today – this is because our servers are being overwhelmed. We’re trying to add machines as fast as possible. We’ve stabilized the game for now and look to add more machines this coming week. Your mafias should all be back to normal now.
  • partial page loads – this is also related to the servers being overwhelmed. In the past, the app would have white screened, now it does partial loads. This issue will also be resolved when we add the servers next week.
  • lost items/progress – This was caused when we rolled out our performance upgrade. People who had the app open while we rolled out the upgrade had their accounts screwed up. In order to fix them, we had to revert them to whatever the most recent backup point was. We gave out items to anyone that we had to rollback to make up for the lost progress.

Major platform changes where you have millions of users online engaging with your game is not easy. No matter how good your QA, it’s darn near impossible to recreate a live scenario where as soon as you roll out, millions are hitting your servers. Equally daunting is finding a good time to have your game go off-line for maintenance, especially on a platform like Facebook where there are millions around the globe hitting your game.

For all intents and purposes, the Mafia Wars platform, launched over a year ago, was probably never developed for the sheer number of users it now supports – a number that has doubled in the last six months and would never have been predicted by even the most bullish of those in the social game space at the time Mafia Wars was developed. There are similar issues with Café World, the number two game on Facebook with nearly 10 million daily active users a day that has seen its numbers drop in the last week as well:

zynga-cafe-world-dau-drop-20091215

Users continue to complain of poor load times or the sim slowing down PC performance when left in the background. Again, this game was produced with some learnings from FarmVille, but the sim is much more animated than the typical farm.

With FarmVille, Zynga took steps to lessen the load on its servers, reducing animations and smooth effects (the introduction of chicken coops and dairy barns as well as storage is also a keen way to push more of the animated items off the screen). Now as Café World creaks under the load, there are small tweaks in place to simplify the animations (sparkling animation over finished dishes seems a bit less intense, pop-ups around dishes are less animated). It’s no wonder that they have yet to launch basic achievements in the game, fearing the impact of more users on a more robust game.

And that also puts in some interesting questions around other recent games like FishVille and PetVille – after fast growth, both games have tailed off a bit. Surely Zynga can advertise and drive more users, so are they holding off in part to ensure that the games can scale?

While Mafia Wars users may fret, you have to give props to Zynga for focusing on focusing resources explicitly to improve their infrastructure. The fallout from trying to fix Mafia Wars when it already is so hug only underscores the need to build your platform in such a way to make it easily extensible and with the option to add servers quickly on the fly when your estimates of traffic are completely blown away.

One thought on “Mafia Wars Gets Whacked: The Woes of Revamping the Infrastructure of a Hit”

Leave a Reply

Your email address will not be published.