What Is Automated Sentiment Analysis Good For?
March 6th, 2010If you’re coming from the post Is Sentiment Analysis an 80% Technology? you may want to continue directly to my post written partially in response to Seth:
Why Sentiment Analysis Sucks for Social Media Monitoring?
Or continue to read the original post that initiated the debate.
———
To save you the trouble of reading further (and few minutes of your time), the answer is “not much”.
Leading providers such as Sysomos and Radian6 estimate their automated sentiment analysis and scoring system to be 80% accurate. That sounds quite good, right?
It doesn’t at all.
20% difference statistically is huge and comes with an array of problems. Think about a situation in which you’re comparing something with 59% positive sentiment to something that has 65% positive sentiment. That’s less than 10% difference. In other words, whenever you have a situation where you’re looking at current data without long historical trend as a reference (for example the Oscars), these types of numbers are completely useless. Almost exclusively, the numbers sentiment vendors provide have differences within the 20% range.
For example, if [A] has positive sentiment of 50% and [B] has positive sentiment of 60%, with 80% accuracy this could mean that [A] is anything between 40%-60% and [B] is anything between 48% and 72%. The only thing this tells us is that statistically speaking [B] might be seen as more positive, but then again, it says the same thing about [A]!
From the below graphic you can see how the benchmark value with 59% positive sentiment can change the whole graph with just 20% variation. So it could be anything from completely positive to quite negative.
Statistics are wonderful for BS.
When you come from the analytics industry and start developing tools for the analytics industry, some things are clear from the get-go. The fact that customers have been teached to ask for sentiment scoring (thank you very much early snake oil peddlers) doesn’t mean the vendors should invest R&D in it. When the ticker feature became available to websites in 96-97, everyone wanted it. Hmmm, come to think of it now, maybe that’s why we struggled with revenues back in the day, after all we refused tickers.
To be a pioneer in an industry is a tremendous responsibility. It’s like teaching a child. If you teach that red is green and green is blue, then that is what the child will learn. If other adults reinforce this message, it becomes the common reality. If the vendor’s teach the market that sentiment is the way to go, then that is what the customers will expect thus forcing future vendors in to a situation where being competitive means doing sentiment better. In other words, wasting R&D resources in trying to fix something that is broken (sentiment analysis) instead of looking at what the customer is really looking for. Customer is always looking for the same thing, make more money. So this model is quite simple.
Brands exist in order to make money, that is the harsh reality we live in. But it’s also a very workable reality from the vendor perspective. Sentiment is nice to know, but up until today I haven’t heard a single commercial application for it. Commercial application in this case means an application that serves the purpose of earning more money for someone else than the vendor of the application.
The question remains:
“How does knowing the net sentiment score help me to drive more commerce?”.
I think a huge problem with the acceptance of sentiment analysis is that many do not think about what a 20% error rate (at best) really means. Your example goes a long way in demonstrating the importance of accuracy. While we see value in sentiment if coded correctly (via real people), it is a surface metric. The real insight comes from understanding *why* sentiment is positive or negative, especially in relation to the competition. Excellent post.
Thanks for sharing, Mike.
From Mike Layton, on March 8, 2010.