If you’re coming from the post Is Sentiment Analysis an 80% Technology? you may want to continue directly to my post written partially in response to Seth:

Why Sentiment Analysis Sucks for Social Media Monitoring?

Or continue to read the original post that initiated the debate.

———

To save you the trouble of reading further (and few minutes of your time), the answer is “not much”.

Leading providers such as Sysomos and Radian6 estimate their automated sentiment analysis and scoring system to be 80% accurate. That sounds quite good, right?

It doesn’t at all.

20% difference statistically is huge and comes with an array of problems. Think about a situation in which you’re comparing something with 59% positive sentiment to something that has 65% positive sentiment. That’s less than 10% difference. In other words, whenever you have a situation where you’re looking at current data without long historical trend as a reference (for example the Oscars), these types of numbers are completely useless. Almost exclusively, the numbers sentiment vendors provide have differences within the 20% range.

For example, if [A] has positive sentiment of 50% and [B] has positive sentiment of 60%, with 80% accuracy this could mean that [A] is anything between 40%-60% and [B] is anything between 48% and 72%. The only thing this tells us is that statistically speaking [B] might be seen as more positive, but then again, it says the same thing about [A]!

From the below graphic you can see how the benchmark value with 59% positive sentiment can change the whole graph with just 20% variation. So it could be anything from completely positive to quite negative.

Possible scenarios with 20% accuracy

Statistics are wonderful for BS.

When you come from the analytics industry and start developing tools for the analytics industry, some things are clear from the get-go. The fact that customers have been teached to ask for sentiment scoring (thank you very much early snake oil peddlers) doesn’t mean the vendors should invest R&D in it. When the ticker feature became available to websites in 96-97, everyone wanted it. Hmmm, come to think of it now, maybe that’s why we struggled with revenues back in the day, after all we refused tickers.

To be a pioneer in an industry is a tremendous responsibility. It’s like teaching a child. If you teach that red is green and green is blue, then that is what the child will learn. If other adults reinforce this message, it becomes the common reality. If the vendor’s teach the market that sentiment is the way to go, then that is what the customers will expect thus forcing future vendors in to a situation where being competitive means doing sentiment better. In other words, wasting R&D resources in trying to fix something that is broken (sentiment analysis) instead of looking at what the customer is really looking for. Customer is always looking for the same thing, make more money. So this model is quite simple.

Brands exist in order to make money, that is the harsh reality we live in. But it’s also a very workable reality from the vendor perspective. Sentiment is nice to know, but up until today I haven’t heard a single commercial application for it. Commercial application in this case means an application that serves the purpose of earning more money for someone  else than the vendor of the application.

The question remains:

“How does knowing the net sentiment score help me to drive more commerce?”.

Connect with Me on Twitter!

Comments (8)

I think a huge problem with the acceptance of sentiment analysis is that many do not think about what a 20% error rate (at best) really means. Your example goes a long way in demonstrating the importance of accuracy. While we see value in sentiment if coded correctly (via real people), it is a surface metric. The real insight comes from understanding *why* sentiment is positive or negative, especially in relation to the competition. Excellent post.

Thanks for sharing, Mike.

From Mike Layton, on March 8, 2010.

Hi Mikko -

Some great thoughts in here. And it’s part of the reason why we haven’t hammered home sentiment as the end-all-be-all of social media analysis, because, well, it can’t be.

Sentiment can have broad applications for things like trends, but if and ONLY if a business is committed to putting a human brain behind the first pass automated results. That means that only a person can understand and interpret things like sarcasm, slang, and other nuances of the human language.

And no matter HOW good the technologies get, there’s no one other than a human that can connect the sentiment trend data to the business goals they’ve set. 75% positive reactions to our product means…what, exactly? What are we going to DO with that information? That’s really the key for applying ANY piece of data, and sentiment indicators are no different in that regard.

Thanks for some thought provoking dialogue. Always happy to chat further.

Cheers,
Amber Naslund, Radian6
@ambercadabra

From Amber Naslund, on March 8, 2010.

Thanks for asking the tough questions.

I agree with Amber on this one. First of all, sentiment should be used to take a pulse, and never as the sole indicator of brand health. Secondly, it will give you an fairly accurate net sentiment score, but if you want to know the sentiment of an article, you should just read it. No automated tool will ever be better than a human, because our speech is way too nuanced (two people agree about sentiment only 79% of the time, so no automated tool can be better than that). Putting a human brain behind it to interpret, and then engage appropriately, is the name of the game.

Maria Ogneva
Biz360
@themaria @biz360

From Maria Ogneva, on March 8, 2010.

I did not read the first link but my analysis is from the above post only.

I totally disagree to the fact that sentiment analysis is having an accuracy of only 80%. Its way more higher. The real problem is not the accuracy but the social indifference of the business to accept this technology. When an marketing agency(manual) gives analysis they (clients accept) but the same thing when a machine does they petrify it. Why?

When OCR (optical character recognition) came in 60’s, everybody said it wouldnt work. But right now entire US postal service works on it. And to add some more info – the system is only 97% accurate, meaning for every 100 mails, 3 mails do not reach their intended recipient. And still I see the people using it. SA is just novice compared to OCR, may be 7 yrs or so – give it some time, its surely going to be buck maker.

From Balamurali A R, on April 9, 2010.

[...] The original post that initiated the conversation  why sentiment analysis sucks for social media monitoring (attempt 1) [...]

From Bruce, on May 19, 2010.

Thank you for a dose of common sense here! Sentiment analysis is such a f***ing buzzword. You know, with only two polarities (+/−), 80% should not be compared to 0%, it should be compared to 50%.

Also ask yourself this: what’s the likelihood that they get 5 out of 5 correct? That’s

.8^5 = 0.32768

or compared to random guessing,

.3^5 = 0.00243

Do people not know Bayesian statistics? Even 99% correct is not that good of a score in some situations.

From human mathematics, on September 22, 2011.

Hello,

I salute you!
I read your article with great interest, and re-posted a link on LinkedIn.

I would like to invite you to read this
http://seojournalist.co/henry-kissinger-vs-sentiment-analysis/
and the follow-up/reply
http://seojournalist.co/henry-kissingers-sentiments-are-not-an-exception/

Good science means stopping bad science from infecting the minds of the gullible!

Best Regards,
Walid Saba

From Walid Saba, on November 29, 2013.

Leave a Comment