Tag Archives: Sentiment

Casual Games, Social Media

A Funny Thing Happened on the Way to Measuring the Forum: Radian6 Workflow and the Missing Sentiment

3 Comments

If you’ve ever run forums around your product, you probably have a good head-start on managing social media –listening, responding, and managing forum posts is not so different from listening to broader media like Twitter, blogs and Facebook Fan Pages. While my team managed our forums well while I was at online multiplayer soccer game Power Soccer, it was difficult to get an overall sentiment of the community, identify hot buttons, and be able to benchmark how we were doing.

I’ve been playing with different social media listening/measurement/operational tools recently and came up with some interesting ways to get a better feel around your forum posts using the Radian6 dashboards. As an example, let’s look at two of the biggest social gaming providers, Zynga and Playfish. Zynga’s games include Texas HoldEm Poker, Word Challenge and Mafia Wars while Playfish scored with Who’s Got the Biggest Brain, Pet Society and just-launched Restaurant City. Below you can see the trending of some of these games’ popularity from Facebook Lexicon (MafiaWars is on quite the upward trajectory):

Conceptually, Radian6 (through BoardReader.com) can extract the number of posts and what is being talked about within the forums, primarily by the number of words used. Here is an example of the “Word Clouds” for just forum posts about Playfish and Zynga over the last 30 days:

Now how to make something out of those clouds is the question, and it’s not very straight forward. You have to weed out things that are part of standard replies (things like “forum”, “quote”, “originally”) and other things are very game specific references (with Zynga’s mafia wars you’d expect things like “hit” or “hitlist” or “attack” or “fight” whereas with Playfish’s Pet Society you’d see “cat” and “owl”). Ideally you’d like to filter out some of these known words to dive deeper into sentiment, but Radian6 doesn’t offer this capability at this time. [Similarly, it would be great to “stem” words, so that cheat, cheats, cheater, cheating would all be classified as “cheat” and you’d be able to see how big that sentiment is within your community. Today, Radian6 only looks at exact word matches.]

So what are the subjective terms you can find that would seem to infer positive or negative sentiment:

  • Playfish: Sentiments are generally more positive than negative: fun, love, nice, thank, versus bad; some mentions of “bought”, “items”, “coins” imply they are talking the things they’ve bought for their characters.
  • Zynga: Not a lot of sentiment-specific words on display here, just “good” and “lost” (diving in on that topic with Raidan6 you can jump to a “River of News” to see each post and infer what is being talked about – in this case it is lost chips or items which I’d classify as a negative customer experience); what DOES show up is that mentions of level(s), point(s), and stats points to a much more competitive user base which is completely logical given games like Poker and Mafia Wars.

In a very rudimentary analysis, it looks like the Playfish forum members are slightly more happy, but again this may be more about the audiences for the games in each community where Pet Society is more social and casual with a community posting 6x as much as the more competitive/cut-throat community on Zynga. While competitive comparisons of forums are interesting, I believe you are better served tracking and improving user sentiment within your own forum.

Word clouds are a poor way of determining overall community sentiment and should be used to identify hot buttons (like the example above where we dived in on the word “lost” and were able to highlight an issue where users were losing poker chips). Radian6 delivers a great work flow tool to do this: once you drill down from these clouds to the customer conversations around that word, you can mark that customer for some sort of action (either assigning the post for immediate response or flagging issues using special tags so you can begin prioritizing potential bug fixes or features required).

Workflow is an integral part of operations, and you can measure the number of tickets and responses you provide, but that doesn’t really ensure you are improving the customer experience. Only by measuring and tracking sentiment can you tell if you are making headway in improving how your customers feel about your brand.

To get a sentiment reading of the forums, Radian6 currently only allows you to hand-code each post and mark them as positive or negative – such a burdensome task that it’s really a non-starter for most marketers. Automating the sentiment categorization task is something Crimson Hexagon has made their point of differentiation (see why this is critical for marketers), while Radian6 noted that automated classification of post sentiment is scheduled for July of this year. So we wait, but relish the thought that the technology (ideally some combination of Radian6 tools and Crimson Hexagon methodology) is nearly here.

Measuring Social Media

Different Input, Different Results: Is it Real-Time User Sentiment without Facebook?

Leave a comment

While I was blogging about the huge negative sentiment in the number of posts for Adam Lambert just before the American Idol results were announced, another company was using much better methodology, but showing a much smaller backlash in the making. Crimson Hexagon was featured on CNN (link to video) and showed true sentiment for the candidates – a lead for Adam Lambert over Kris Allen on a positive level, and a scant 3% negative impression for Adam Lambert with no real backlash at all for Kris.

Why did I, using a very rudimentary love/hate modifier with Vitrue’s Social Media Index, see a huge backlash in relative terms for Adam vs Kris while Crimson Hexagon saw a much smaller difference?

For one, my modifiers were rudimentary – there are many more ways to show whether you are strongly for, for, against or strongly against something. My sample was only a very small slice of the posts out there and the data in the Vitrue Social Media Index (SMI) isn’t designed to show positives and negatives.

But what the SMI does appear to have, that Crimson Hexagon does not, is Facebook data.

I’m a big fan of Crimson Hexagon’s methodology to defining user sentiment (see why I think automating post sentiment is such a critical component of managing social media). But without Facebook status and wall feeds as part of its data input, I believe it misses some critical understanding of real-time sentiment among the masses.

Yes, Twitter is part of the micro-blogging component of real-time sentiment, but I don’t think users are as addicted or connected to the medium as they are with Facebook. In general, I believe you say things among your friends (e.g. among Facebook circle and within the walled garden) that you may not want to broadcast from the top of a hill for all to hear (e.g. Twitter). I would also argue that Twitter is much more about gathering data (their founders mentioned it as more of an info dissemination tool) and reading for the great wave of newbies than it is about truly interacting with other users.

Thus to really get a feeling for real-time sentiment, Facebook is a critical component. I believe lacking the Facebook feeds contributed to Crimson Hexagon’s negative measures being more muted than what I saw with the Vitrue SMI, and illustrates why marketers need to understand the underlying inputs in any system or technology they choose to leverage to view real-time sentiment.

Measuring Social Media

Automating Classification of User Sentiment is the Key to Unlocking Social Media

1 Comment

Rush Limbaugh has 33% more buzz online than Jay Leno according to the Vitrue Social Media Index. Social game company Zynga has tweets by posters with 3 times as many followers as their competitor Playfish according to the data over the last 30 days from Radian6. But it doesn’t really mean anything unless you know what percentage of that buzz or those tweets are positive or negative.

Sentiment is the guiding light that helps marketers and their organization know what to do with the mountain of social media that is growing exponentially through Facebook and Twitter. We need to understand our current positive to negative ratios, drill down to uncover what’s driving it, then put programs in place to mitigate the negative and foster growth of the positive. Just like we determine the ROI on capital improvements, with an accurate measurement of positive and negative sentiment we can measure the return on investing in programs to improve consumer sentiment about our brands.

So the vision thing is great, but the reality of going through thousands of posts and tweets and determining sentiment is the biggest impediment facing marketers and their organizations to moving forward. The industry average in being able to automate this process of categorizing the sentiment of user posts is about 60%, based on a conversation with Radian6.

Radian6 is a great workflow tool, but today they only offer users the ability to hand code the sentiment of each post. Their goal is to introduce automated sentiment attribution into the Radian6 dashboard this summer to get to 70 or 75% accuracy.

Crimson Hexagon has shown through research by co-founder and Harvard Professor Gary King that its approach to automatically categorizing the percentage of posts in blogs is higher than hand-coding or strictly counting the number of words. “Crimson Hexagon doesn’t count words, which can mislead; it amplifies human judgment to give the percentage in each category accurately,” noted King in a recent tweet. One of King’s colleagues noted that Crimson Hexagon is close to 80-88% accuracy for positives and negatives using their approach.

Very few marketers and organizations will spend the resources to hand-code responses (in fact King’s research suggests that one shouldn’t and typically see diminishing returns after 500). We will more and more rely on automated tools to do this work for us. Thus when the time comes to pick vendors, agencies and tools to help us measure sentiment, it is critical for us to understand the underlying data, methodology and resulting accuracy rates.

To date Crimson Hexagon’s methodology seems to provide the most promise. What other tools are you using to identify positive and negative sentiment of your brand and what is their underlying methodology?