Big Data Comes with Big Challenges
"If I'd asked my customers what they wanted, they'd have said a faster horse."
– attributed to Henry Ford
"If you can't measure it, you can't manage it."
– Peter Drucker
"Not everything that can be counted counts, and not everything that counts can be counted."
– Albert Einstein
If you Google the term big data, you will get over 5 million hits. While this figure is nowhere near top-tier buzzwords such as social media (320 million), it is about the same as credit default swaps, and nearly five times that for predator drones. The potential value in analyzing large, and increasingly unstructured, data sets has clearly resonated with the business press, IT suppliers and a growing number of customer organizations.
But as the three quotes above suggest, even great minds can have very different views regarding the relative importance of factual information, and the tension between data and intuition has always been with us. The career of Steve Jobs shows that trusting one's own judgment, instincts and aesthetic sense is often the best way to pursue breakthrough innovation. That's why he has been so frequently associated with the Henry Ford quote (even if there is no evidence that Ford ever actually said it).
Of course, most of us don't pursue breakthrough innovations very often; we are mainly focused on getting better at what we do. When improvement is the goal, information can be invaluable and Drucker's view rules. Using data to improve measurement and management is the main purpose of the big data movement. (We'll get to Einstein's quote later.)
From an LEF research perspective, there have been two recent big data developments that we want to call attention to in this commentary. First, our sister organization, the LEF Technology Programme has released The Data rEvolution, its 75-page survey of the big data opportunity, highlighting today's impressive range of data-intensive technologies, applications and future possibilities. This report is now being distributed to LEF Executive Programme clients. Second, we recently held a research focus group in London on the relationship between IT and Marketing, where the great majority of the participants agreed that their firms are struggling to become truly data-driven. Apparently, the gap between the promise and the reality of big data usage is still wide.