Big Data: Too Big? Or, Too Noise?

Every day, 2.5 quintillion bytes of data are created and 90% of the data in the world today was created within the past two years. The increasing volume and details of information captured by enterprises, the rise of multimedia, social media, and the Internet fuel exponential growth in data for the foreseeable future.

Interestingly, two thought leaders in customer management and strategy, Wim Rampen and Esteban Kolsky , also discussed about Big Data. Wim Rampen writes, ” There is a lot of noise, and despite improvements in Social Media monitoring, analytics tools/solutions and what have you, we will need a lot more powerful tools to connect the dots and see patterns. We may need Watson-like technology to automate these processes and then still the outcome is not sure.” Further, in an isightful post, Esteban Kolsky, says, ” Indeed, the greater challenge to organizations is not how to manage Big Data, rather how to separate data from noise and just handle data and discard noise…You don’t need a new analytics strategy, you need a new filtering strategy. ”

So, what is Big Data?

Big data is a term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target currently ranging from a few dozen terabytes to many petabytes of data in a single data set.” –Wikipedia

Big data are so large that they become awkward to work with using on-hand database management tools. Difficulties include capture, storage, search, sharing, analytics, and visualizing. The is an impending need to derive meaningful information by analyzing this huge amount of data, which is in turn driving development of technology and business solutions to deliver data strategies for diverse use cases.

According to research by MGI and McKinsey’s Business Technology Office, “… analyzing large data sets will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus.” The research offers seven key insights:

1. Data have swept into every industry and business function and are now an important factor of production, alongside labor and capital.

2. There are five broad ways in which using big data can create value.

First, big data can unlock significant value by making information transparent and usable at much higher frequency.

Second, as organizations create and store more transactional data in digital form, they can collect more accurate and detailed performance information on everything ….
Third, big data allows ever-narrower segmentation of customers and therefore much more precisely tailored products or services.
Fourth, sophisticated analytics can substantially improve decision-making.
Finally, big data can be used to improve the development of the next generation of products and services.

3. The use of big data will become a key basis of competition and growth for individual firms.

4. The use of big data will underpin new waves of productivity growth and consumer surplus.

5. While the use of big data will matter across sectors, some sectors are set for greater gains….The computer and electronic products and information sectors, as well as finance and insurance, and government are poised to gain substantially from the use of big data.

6. There will be a shortage of talent necessary for organizations to take advantage of big data. By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions.

7. Several issues will have to be addressed to capture the full potential of big data… Organizations need not only to put the right talent and technology in place but also structure workflows and incentives to optimize the use of big data.