Common Challenges with Interpreting Big Data (and How to Fix Them)

There's a lot of science to Big Data. But to derive real value from it, you still need a human touch--especially when it comes to interpretation.

When you have a pile of data, many analysts feel an urge to dive right in. Run queries and try to tease out predictive models right away—without a proper goal. Others find the variety and scale of available data confusing, and think they won’t get any meaningful results.

Take a step back. This process has many challenges, which can hamper you if you don’t take the time to plan for them.

Let's look at 3 such challenges. Along with fixes you can put in place easily.  Save yourself time, & potential ‘big’ data mishaps later on!

1. Not Verifying the Data’s Usefulness

Collecting data is great.  It gets easier to do by the day. So easy, in fact, that we can collect terabytes of 100% useless data before we know it!

How do we make sure data is useful then?  Compare its origin, its freshness, and its workflow relevancy to your existing business goals.

Ask yourself, “Which business/research decision will this analysis help us reach?  How will we derive value from the result, and apply it to that goal?  How can we learn from the experience, in order to re-apply it to future goals?”

Let THAT serve as your usefulness standard.  

In 2014, Western Union started using Big Data to help it stop wire fraud.  Western Union processes 2.4 transactions every second.  They have to move money, look for fraudulent transactions, block those transactions, provide a good user experience for legitimate customers, and keep customers coming back despite many online alternatives. That makes a huge amount of data to sort through.  Data they must make useful.

They used a Hadoop solution to sort through the freshest data, looking for fraud patterns.  Running queries through the available data sets makes spotting fraudulent transactions easier (and accidentally flagging legitimate transactions less likely).  CIO David Thompson said, after they began using Hadoop, “I have data that can answer the questions that the business side has been asking for 10 years. I can respond better than ever before.”

Several police departments, including the LAPD, now use a Big Data solution called PredPol to predict crime.  PredPol software collects data about a specific area, and verifies the data’s usefulness.  Police then run queries & make predictions about activity in their neighborhoods.

In one of LA’s divisions, crime went down 13% in the 4 months following a 2014 rollout.  Burglaries, car break-ins, property destruction...down 13% just from one tool helping police do their job.  That’s some useful data.

The Fix: Choose an interpretation technology that supports flexible Data Discovery. Discovery gives you the ability to ask different questions on the same data set as well as incorporating other data sets on the fly. If you find your data isn’t useful, you can add in new data sets & run fresh queries right away. This is especially important given all the disparate sources & structures of new data out there.

2. Data Sets are Too Varied/Irrelevant

Say your CIO wants to start informing all major decisions with Big Data.  To start off, you have the following data sets available:

• Feedback from customers in a database.  Mostly text, with some images.
• Unstructured data from research studies (format varies between text, audio & video).
• Streaming data from social media feeds (not filtered).

All viable data sets, even in their disparate formats.  You might think the next action is loading all of the data & starting on queries.

Hold on.  The better thing to do is ask yourself:  “Does it make sense to integrate all of these data sets before beginning my data discovery?  Are they all relevant to the decision before us?”

Let’s take a real-world business goal as an example.  The University of Montana wanted to help its patients, by identifying which ones were at risk for epilepsy.  It had a huge data set - 50,000 hours of video data from those patients.

The data was unstructured, but completely relevant.  And accessible—they could pick any video & watch it.  But manually combing through 50,000 hours worth of video?  Not feasible.

Instead, they turned to Big Data for insightful analysis of the video.  What was their goal?  Improving the efficiency of their current research.  The result?  An 80% improvement in research time.  The University supercharged its ability to find & treat at-risk patients.

Now imagine what would have happened if they dumped in data from an irrelevant department.  Cancer treatment success rates, let’s say.  They would have polluted the analytics and had to start over.  Losing even more time.

The Fix:  Only include those data sets which you deem relevant to the immediate business goals.  Just because you have 8 data sets to use doesn't mean you use all 8 at once.

If your data sources are relevant to the goal, the results will help you sharpen your knowledge of that goal.  If you find they don’t, you likely have irrelevant data mixed in.

Not sure which data sets are relevant to start with?  Try this as a qualifier: “What did I intend to do with this data set before I collected it?”

Also, make sure to use a Data Discovery technology that lets you fuse disparate types of data (e.g., real-time data, machine/sensor data, social media feeds, data streams, traditional structured data, etc.) on the fly.  This way you can identify whether results are relevant or not, and adjust data sets accordingly.  

Visual analytics work well here—your eyes can pick out skewed results faster visually than by reading.

3. Managing Unstructured Data

“What about all the various formats in unstructured data?  I don't even know where to start fusing these together!”

The future belongs to unstructured data.  Its sheer growth vector is intimidating—an estimated 40% per year!

It grows so fast because of all the formats in which you find it.  Emails, plain text, flat files, web logs, machine & sensor data, geolocation info, clickstreams...with more formats appearing all the time!

Its volume is already up to more than 1,600 exabytes (80% of the world’s total data, by some estimates).  By 2020, IDC estimates that number will reach 40,000 exabytes.

1. Preparing the data correctly, and
2. Speed of analysis.

Thanks to technical innovations like Hadoop, NoSQL & Apache Spark open source technologies, unstructured data no longer complicates Big Data collection, storage, and preparation.  But it can still eat away your time.

First you have to prepare the data.  Assessing (as in our above challenges), formatting, and fusing.  You can do this yourself - some analysts prefer to, thinking of data fusion as more an art than a science.  But this is another step, and one that slows down deriving results from the data.

The Fix: Skip the extra step of separate data fusion.  Use an analytics solution that does the fusion for you, letting you get to query results fast.  This saves you time, and allows you to ride the wave of innovation in Big Data.

Unstructured data’s growth will take the entire Big Data field into new horizons.  Some of the solutions in use now won’t survive...unless they’re already built to grow & evolve along with unstructured data.  Keep the future in mind when selecting your tools.

Remember: Big Data is a Journey

Interpreting Big Data is the human part of data-driven business. It's when you look at the “How” (the results of Big Data analysis) and ask “Why?”

Tackle interpretation challenges as a balance between value & time.  You won’t solve every question at one go, but losing sight of your goals or working with irrelevant data simply burns time.  

Big Data analytics is a journey where you make incremental progress - set yourself up right ahead of time, and you’ll gain priceless insight into the next thing customers need from you.