How to interpret large data

"From the dawn of civilisation until 2003, humankind generated five exabytes of data. Now we produce five exabytes every two days... and the pace is accelerating" - Eric Schmidt, Executive Chairman, Google.

If you’ve even slightly interest in marketing or statistics, there’s a good chance you’ve heard the term ‘big data’. There’s an equally good chance, however (and despite that it’s been around for years now), that you mightn’t know exactly what it means.

Big data is a term used to describe datasets that traditional data applications aren’t capable of processing. The term is a testament to our hyper-connected digital world, responsible for the widespread and exponential proliferation of knowledge, in every shape and form. In other words, big data is a (useful) buzz term we made up to name and emphasise just how much information there actually is floating about in our modern world.

To give an idea of the scales of information we’re talking about, consider this. In 2012, according to research firm IDC, we created or replicated about 1.8 zettabytes (ZB) of data. Just a few years ago, we hadn’t even invented the word zettabyte. Every day, for instance, we send approximately 300 billion emails and share one billion items on Facebook. Every minute we post 170,000 tweets to Twitter, 3,000 photos to Flickr and 48 hours of video to YouTube. That’s not just big data; that’s HUGE data.

Utilising & Understanding Big Data

It’s true that big data is extraordinarily useful. Big data gives us an intimate insight into how people are interacting with their worlds. It tells us what trends are shaping, collapsing, how people are spending their time and their money. What people are doing, and even why; it is effectively a reflection of consumers’ behaviour and patterns of thought, bottled up and mapped out digitally. It’s gathered from innumerable sources – various devices, various software, various platforms. It’s BIG data.

Obviously, organisations (as well as small business owners and marketers) are looking for ways to make the most of these mountains and mountains of information. It’s one thing to have big data; utilising it, however, is a completely different story. Let's investigate som of the more well know uses for big data, globally.

Understanding Your Customers

Customer profiling is probably the most recognised use for big data, and good reason. Big data allows organisations the opportunity to better understand their customers, their purchasing patterns, seasonal spending, behaviours and product preferences. Businesses have become increasingly excited about this information, looking to develop greater data sets through browser logs, text analytics, sensor data and social media to gather a more complete and comprehensive image of their customer segments. The big objective, in many cases, is to create predictive models.

Remeber Target's 'pregnant teenager' mishap. When the U.S. retailer was able to very accurately predict when one of their customers was expecting a baby, unexpectedly notifying her family with tailored sales collateral sent to her home before her family was even aware of the expectancy!

Target isn't the only one though capitalise on big data. Other retail and telecommunications suppliers have been able to use big data to assess seasonal spending and customer churn. Insurance companies are using big data to better understand their customers behaviour. It has even been suggested that presidential election campaigns in the US are heavily influenced by big data, as political groups look at data patterns to predict policy favouritism and electrical approvement for specified states and regions.

Optimising Business Processes

Big data is allowing organisation to optimise their business process in a number of distinct and significant ways.

The Retail and Hospitality sectors have been able to significantly improve their stock supply predictions, now able to cater for seasonal changes better than ever before with data taken from web searches, trends analysis, weather forecasts and even social media.

HR business processes have also been improved using big data analytics. This includes the optimisation of talent recruitment (think Moneyball) as well as the measurement of company culture and staff engagement.

Another business process seeing a lot of big data analytics is supply chain or delivery route optimisation. Here, geographic positioning and radio frequency identification sensors are used to track goods or delivery vehicles and optimise routes by integrating live traffic data.

Driving Machinery Performance

Big data is even helping to improve efficiencies for machines, helping to make these devices 'smarter' and improving automation in the process. Google has been able to capitalise on their rich source of data, now using it to help operate its mapping with self-driving car. Other organisations such as Toyota have been able to integrate big data in their cars, such as the Prius which uses a camera, GPS as well as powerful sensors to safely drive on the road without the intervention of humans.

Big data tools are also being used to optimise and improve energy production through energy grids, which use big data taken from smart metres. Companies are even using big data to help improve the optimisation and performance of computers in data warehouses.

Big Data – Even Bigger Challenges

For the average person and business without access to super fast and super smart computers and algorithms, making sense of big data might seem an impossible task. It would be like walking into a gigantic library, whose books aren’t ordered, and being asked to arrange them, first by genre, then by sub-genre, then by author. Not an enviable task by any stretch of the mind.

The truth of the matter is a little bit different. Although big data does seem to exist in an impenetrable box, locked away on a shelf so high that no mortal can reach it, it’s not. Well, not completely.

Business and organisations are now learning just how important and effective access to big data is. This means that more and more businesses, marketers, and statistical analysts are working on decoding it, thereby making it accessible to all of us. But that’s as far as I can go; the rest is up to you.