August 31, 2012 | By Rey Villar
Every organization is absolutely drowning in data. Yours is no exception. We casually talk about data in numbers humbling to astronomers. Data-driven decision making appears to have won the battle over intuition. Moore was right about predicting the density of storage, but does this increased capacity automatically translate to better business intelligence?
I would suggest that it’s not about the data.
Don’t get me wrong. Data enriches and informs. It brings profits to the wise, efficiency to the observant, power to the wielding. Data describes many events with God-like precision. Gordon Gekko could update his money mantra for the 21st century: data is good.
But data doesn’t always remain good and usage requires caution.
Data offers a confidence level that can be difficult to dislodge – especially when convincing but wrong.
Data can be misinterpreted or misused. It can ignore the mentality of the herd or blind us to the possibility of a black swan or ‘six sigma’ event. Data can lead us down inescapable analytical rabbit holes, paralyze us with options, render invisibility to the subtle, incomprehensibility to the complex. Data can go bad as well as do badly.
Data can be bad.
Sometimes, we can’t be sure if data is friend or foe.
Much to the detriment of IT professionals, data doesn’t intrinsically know if and when and why and how it has gone bad. Most importantly, data doesn’t know that it has gone bad. Unlike Hal in ‘2001: A Space Odyssey’, our data systems lack self-awareness, guilt or remorse, and have little, if any, capacity for self-healing. Despite half a century of progress, many systems still live within a ‘garbage in, garbage out’ paradigm. As a result, we are compelled to live in a constant state of vigilance. Data can’t be left for long without adult supervision.
Our organizations have responded to the challenge. We’ve applied data quality tools, mastered golden records; installed audit / balance/ controls; governed the tools and processes and data producers and keepers and consumers; applied BPM and IE and a dozen additional frameworks. Still, our data runs amuck, does things that are incomprehensible, misses problems that seem so obvious on discovery. We sense endless missed opportunities, mine for ever more data points, seek yet another data source to discipline. The cherished rosetta stone is but a flat file away! We are enamored with data despite the slipperiness of keeping it clean and suitable for use.
So what’s missing from an all-data, all-the-time world? Human judgment.
Many real-world problems simply cannot be solved with data.
Long before computers collected, aggregated and analyzed our transactional lives, intuition was used to make the call. In many situations intuition is still the favored path for decision making.
A world absent human judgment and intuition is myopic. Many problems require expert thinking and management skills to solve.
Data won’t help prioritize system replacement. Data can’t write your IT strategy, hire the right resources to execute that vision, write a project plan to get there, or drive the daily activities that will ensure progress and success.
Data has difficulty dealing with the counterintuitive, the illogical, the non-linear and non-scalar. Fuzzy logic aside, there are still domains where gray matter rules over silicon.
As data drives more deeply into every aspect of our lives, the split grows between those who leverage data for insights and those who expect data and data systems to solve all the tough problems. Expect data to provide insights to supplement, not supplant, the limits of our thinking.
Business intelligence succeeds where disciplined, high-quality data is applied with expert judgment and old-fashioned common sense.
It’s not about the data.