A Funny Thing Happened on the Way to Measuring the Forum: Radian6 Workflow and the Missing Sentiment

If you’ve ever run forums around your product, you probably have a good head-start on managing social media –listening, responding, and managing forum posts is not so different from listening to broader media like Twitter, blogs and Facebook Fan Pages. While my team managed our forums well while I was at online multiplayer soccer game Power Soccer, it was difficult to get an overall sentiment of the community, identify hot buttons, and be able to benchmark how we were doing.

I’ve been playing with different social media listening/measurement/operational tools recently and came up with some interesting ways to get a better feel around your forum posts using the Radian6 dashboards. As an example, let’s look at two of the biggest social gaming providers, Zynga and Playfish. Zynga’s games include Texas HoldEm Poker, Word Challenge and Mafia Wars while Playfish scored with Who’s Got the Biggest Brain, Pet Society and just-launched Restaurant City. Below you can see the trending of some of these games’ popularity from Facebook Lexicon (MafiaWars is on quite the upward trajectory):

Conceptually, Radian6 (through BoardReader.com) can extract the number of posts and what is being talked about within the forums, primarily by the number of words used. Here is an example of the “Word Clouds” for just forum posts about Playfish and Zynga over the last 30 days:

Now how to make something out of those clouds is the question, and it’s not very straight forward. You have to weed out things that are part of standard replies (things like “forum”, “quote”, “originally”) and other things are very game specific references (with Zynga’s mafia wars you’d expect things like “hit” or “hitlist” or “attack” or “fight” whereas with Playfish’s Pet Society you’d see “cat” and “owl”). Ideally you’d like to filter out some of these known words to dive deeper into sentiment, but Radian6 doesn’t offer this capability at this time. [Similarly, it would be great to “stem” words, so that cheat, cheats, cheater, cheating would all be classified as “cheat” and you’d be able to see how big that sentiment is within your community. Today, Radian6 only looks at exact word matches.]

So what are the subjective terms you can find that would seem to infer positive or negative sentiment:

  • Playfish: Sentiments are generally more positive than negative: fun, love, nice, thank, versus bad; some mentions of “bought”, “items”, “coins” imply they are talking the things they’ve bought for their characters.
  • Zynga: Not a lot of sentiment-specific words on display here, just “good” and “lost” (diving in on that topic with Raidan6 you can jump to a “River of News” to see each post and infer what is being talked about – in this case it is lost chips or items which I’d classify as a negative customer experience); what DOES show up is that mentions of level(s), point(s), and stats points to a much more competitive user base which is completely logical given games like Poker and Mafia Wars.

In a very rudimentary analysis, it looks like the Playfish forum members are slightly more happy, but again this may be more about the audiences for the games in each community where Pet Society is more social and casual with a community posting 6x as much as the more competitive/cut-throat community on Zynga. While competitive comparisons of forums are interesting, I believe you are better served tracking and improving user sentiment within your own forum.

Word clouds are a poor way of determining overall community sentiment and should be used to identify hot buttons (like the example above where we dived in on the word “lost” and were able to highlight an issue where users were losing poker chips). Radian6 delivers a great work flow tool to do this: once you drill down from these clouds to the customer conversations around that word, you can mark that customer for some sort of action (either assigning the post for immediate response or flagging issues using special tags so you can begin prioritizing potential bug fixes or features required).

Workflow is an integral part of operations, and you can measure the number of tickets and responses you provide, but that doesn’t really ensure you are improving the customer experience. Only by measuring and tracking sentiment can you tell if you are making headway in improving how your customers feel about your brand.

To get a sentiment reading of the forums, Radian6 currently only allows you to hand-code each post and mark them as positive or negative – such a burdensome task that it’s really a non-starter for most marketers. Automating the sentiment categorization task is something Crimson Hexagon has made their point of differentiation (see why this is critical for marketers), while Radian6 noted that automated classification of post sentiment is scheduled for July of this year. So we wait, but relish the thought that the technology (ideally some combination of Radian6 tools and Crimson Hexagon methodology) is nearly here.

3 thoughts on “A Funny Thing Happened on the Way to Measuring the Forum: Radian6 Workflow and the Missing Sentiment”

  1. Hi Eric,

    Thanks so much for this comprehensive drill-down. You’re right that automated sentiment is coming from us in the next few weeks. The trick of course is that no automated sentiment is yet perfect (we English-speaking humans have a complex language, don’t we?), but it’ll at least give you a head start with a first pass at the information so you can refine it based on your human filters.

    Appreciate some of your other comments and suggestions, and we’re always open to feedback on how to make the platform better. Thanks very much for taking the time to write this up, and please reach out anytime.

    Cheers,
    Amber Naslund
    Director of Community, Radian6
    @ambercadabra

  2. Hello Eric: I agree that word clouds are best used to identify issues rather than sentiment. Amber is also right, machine scoring of sentiment is just not perfect. Our testing has found that humans agree with each other about 85% of the time, machines do a little worse, but that for brands with more than a few dozen mentions a week a directional estimate of sentiment is really, really valuable.

    One important thing to note about sentiment ratings is whether they are about the whole post or the specific thing or person within the post. For instance, a post that says “I usually hate shopping at Target but the Go International line is too cute to resist” is positive about Go International- but not about Target. At Scout Labs we focus on entity specific sentiment, not positive and negative words counts, for this very reason. Users can correct machine estimates if absolute precision is required, and many do, which helps us to find words like “suxxxx!!!!!!” which we believe connotes negative sentiment and makes our dictionary powered algorithms much better over time. Come try it out and see how it works, we offer free trials to one and all.

    Best, Margaret Francis
    VP Product Scout Labs
    info@scoutlabs.com

  3. Good post Eric.

    As my friends above point out, this is tricky stuff. Each of us has very smart people working on the problem of how to infer sentiment from unstructured commentary, and Margaret is right to point out the theoretical limits of any system that approaches this problem through the words themselves.

    Crimson Hexagon is unique in approaching it through numbers, though, or more properly though numerical representations of this same text. Doing so reveals patterns beyond even what is readily discernible by human beings, enabling us not only to diagnose the “black and white” tones of sentiment with unmatched accuracy, but to paint a picture in the many “colors” of a conversation, for each theme, thought or emotion present.

Leave a Reply

Your email address will not be published. Required fields are marked *