Big data danger of confusing signal and noise

Big data can be a distraction. Real life insights are not growing nearly as fast as irrelevant data argues Adam Parker. The solution lies in a more disciplined approach to the sources we trust. By Adam Parker

A recent study by PeerIndex hailed the five members of One Direction as the most influential UK Twitter users ahead of David Cameron and Ed Miliband.

Don’t get me wrong, with a daughter who is now starting to really get into music I’m aware that these five gentleman do indeed have significant potential for influence over a particular demographic.

But the most influential UK Twitter users? If that’s the case maybe they should be running the country.

Image copyright Lissted /Andrew Waugh

To me it feels more like a good example of how a certain approach can mislead PRs and marketers. And the blame must fall somewhere between the data and its interpreters.

So what’s going wrong?

It's clear that our society is generating new data at an ever increasing rate - hitting an estimated 40 zettabytes by 2020 according to tech storage giant EMC.

But for those who hope to find insight from that data, or signals among the noise, there's a scary realisation. The amount of relevant things in the world isn't growing nearly as fast as those irrelevant and distracting data points around them.

Renowned statistician Nate Silver puts it well.

“We're not that much smarter than we used to be, even though we have much more information – and that means the real skill now is learning how to pick out the useful information from all this noise,” he says.

Bad answers are easy to come by. Good questions aren't

This presents a problem for the currently most common “outside in” approach of finding information in this data. In trying to digest the entire universe of data and work back from that towards the important signals, the task becomes harder and harder and the risk of lapsing levels of accuracy becomes greater and greater.

Luckily, there’s an obvious alternative - one that hopefully finds a lateral solution to the challenge Nate implies: an “inside out” approach, starting with the signal you know and using it to find more with just the right measures of relevance.

To effectively find new people that matter, it makes sense to start with who you accept already fits this description. Using their interactions out into the wider world, you can then build your understanding by learning from those relationships.

This is the inherently human approach that savvy PR and marketing professionals have always used. The challenge is how to accelerate and scale this process to match the vast and fast moving social media world we now find ourselves in.

This is where a tool like Lissted comes in. We use technology to supercharge this human approach to finding signal in social noise. The result is accurate predictions of people and conversation that matter, which we hope even Nate might be proud of.

Lissted UK Food and Drink example

When you compare it with the ‘big data’ alternative of trying to listen to every single message out there and then work back, while avoiding distortions by spam or over-weighting the noisiest people, we think the choice is pretty clear.

Unless of course we really do think One Direction should be running the country?

About the Author

Adam Parker is founder of Lissted and Chief Executive of RealWire. He contributed chapters to both of the CIPR’s Share This books and is a member of the CIPR’s Social Media Panel. He is a Chartered accountant and previously worked for PwC’s audit, corporate finance and consultancy practices. You can connect with Adam via Twitter @adparker.