This Is Why You’re Failing at Big Data


By now, anyone who uses data has heard about the three “V”s of big data. Let’s not debate whether there are four or five or 20 “V”s. Whether your concern is millions of marketing research panel records, billions of retailer transactional records, or trillions of website tracking records, anyone who uses data has the same problems. The volume of data increases exponentially, the variety of data keeps increasing, and the speed with which data becomes available is, well, let’s think of the speed of light. These issues alone make dealing with big data a complicated procedure.

Add to that the fact that big data is rife with missing data. It’s incomplete data, and it’s complicated data. It needs specialized analytical tools and specialized analysts. But those problems are also not the reason we’re failing when it comes to big data.

Why are we failing at big data? Well, let’s take a step back to the survey and focus group world that market researchers and insight professionals love so much. When I think back to the last survey I wrote, it too was quite the beast. For just twelve minutes of respondent time, I spent many hours dreaming of, writing, tweaking, rewriting, and retweaking every single question and answer. I pondered every “the,” “or,” “if,” “they,” “you,” and “why.” I argued with myself about the possible ramifications that every single word might have on my results. In every case, I settled on the best solution, not the right solution. In the end, I had a survey that would carefully address every single hypothesis and research objective on my list. This survey was a beauty and the analysis was quick and easy.

Let’s move forward to your big data project. You know, the one where someone dumped a giant —> Read More