Big Data – seeing the wood for the trees

An interesting article on CNet highlighted one of the most talked about topics in the analyst playground at the moment: Big Data. Nokia is now using location-based data and tracking and mapping technologies to gather 21 billion data points per month from drivers. It is claimed that this data will enable Nokia to see how the average driver handles the road, with the intention that, in future, your car will be able to issue warnings about dangerous curves, weather warnings and other road hazards.

With tens of millions of people across the globe now ‘connected’ through billions of sensors and trillions of transactions carried out on a daily basis, it is easier than ever to record an unimaginably scary amount of data. From Set Top Boxes and Electronic Point of Sale to Social media, data is being collected on everybody, on all aspects of life. But is it possible that the sheer mass of data to process and handle will actually hold analysts, and thus businesses, back rather than strengthen their analysis and provide new and useful insights? Can we really trust all of this data to tell us when we should slow down and when it is safe to go faster on our roads?

The proposed benefits of Big Data are pretty straight forward. Being incredibly detailed it provides low cost, highly granular measurements of behaviour. The data does not rely on surveys but on actual behaviour so is likely to be less biased and, due to the sheer volume of information collected, it allows a narrower segmentation of customers leading to more precise targeting of products and services.

However, despite many within the industry hailing the rise of Big Data as a revolution of sorts, some remain sceptical as to how easy this data is to handle and how much easier it will make the work of analysts in the future. More granular data is not always better as it is easy to miss the wood for the trees and Big Data is not necessarily good data - it is essential precision is not confused with accuracy. Most Big Data measures devices, not people, and there is a huge amount of white noise encumbering the data. If this data is not properly aggregated, there is the danger of it being misleading. Bryan Trogdon, entrepreneur and Semantic Web evangelist, said 'Big Data is the new oil. The companies, governments and organisations that are able to mine this resource will have an enormous advantage over those that don't.' However, if Big Data is seen as oil it still needs refining. One of the most time consuming jobs as an analyst is cleaning up data, spotting and solving inconsistencies and anomolies and chasing missing data. The bigger the data, the more of these errors there will be.

Download the article to read more