CONTACT US 224 265 0400 or Email

The Agile Incubator Blog

Are “advanced analytics” really all that advanced?

We have roughly 20+ years of BI/DW projects under our belt – but how satisfied are we at the extent or depth of our reporting and analytic progress?…that the promise of advanced analytics is being realized?…and that we are operating as an agile organization, with the ability to derive meaningful insights for decision-making and taking action? Before approaching this topic, it’s useful to distinguish categories of reporting and then think about the Information Providers (those who create the reports) and Information Consumers (users).

Categories of Analytics

1. Standard Retrospective Reporting: At a minimum, an organization tracks key performance indicators, which serve as leading or lagging measures. Typically the information consumer has a sub-set of selected, pre-defined selection criteria, measurements and metrics available which can be combined and segmented.

2. Exploratory Analytics: Being able to infer patterns is a more advanced technique for deriving insight from data. Power analysts/ report developers serve as the “go-to” folk for the business lines (and typically have a never-ending backlog of requests from the information consumers they support).

These first two categories help us identify exceptions and answer basic questions. They explain the past or current…and they are the primary focus of the BI tools and OLAP available in the market.

We’re no longer content with mere “trends” out of our data. Today’s DWs with huge amounts of data can highlight for us where the exceptions and anomalies reside. The TRUE value out of our data requires statistics, data mining, regression analysis, complex models and algorithms.

3. Predictive Analytics: Ah, this is the place…the BI/DW vision being realized.  It’s where we gain an understanding of complex relationships between data; have the ability to chew through mind-boggling amounts of data; and predict behavior of our consumer.  But it remains neither easy nor intuitive.  A select few of our information providers still roll-up their sleeves and go through the rigor of regression analysis, statistics and probabilities. Twenty years and we’re still reliant on understanding advanced math for getting answers?

So while considering predictive or ‘advanced’ analytics, I come across the interesting link…and am visually awed at the range of presentations of data. YES! –Look at all that data from all these customers across all these years being visually captured and conveyed. But…sadly, improved design in our tools does not mean advanced analytics. To remain competitively viable, we have to be able to deduce important consumer behavior. Querying and reporting do NOT provide this – the volume and complexity of the data means you need more than SQL and your basic BI tool usage.

Advanced Analytics

TDWI lecturer Mark Madsen asserts that advanced analytics is more feasible now because “the drivers behind this are not necessarily new algorithms or better tools, but the availability of computing power and storage at affordable price ranges…caused by the fact that nowadays it is feasible to analyze many gigabytes of data in a near real-time fashion, something that was impossible to do 5 years ago)”. He describes how advanced analytic techniques and tools are out there for all types of data mining usage (and some are free!).  But you have to figure out which tools / techniques (see table below) apply to and are most suitable for your ultimate purpose: Churn/attrition modeling?…customer acquisition?…behavioral advertising?…email targeting?…CLV Modeling?…product recommendation system?…upsell/cross-sell?…click fraud detection?




Pre-defined metrics, measurements & questions – exact-match retrieval Scientific application of math to data collection, analysis and presentation TECHNIQUES: Use of known variables to predict unknown or future values of other variables such as – Linear regression – Neural networks – Decision, regression trees

  • Classification & (vs) clustering
  • Association rules
  • Sequential pattern discovery
  • Logistics regression
  • RFM (Recency – Frequency – Monetary) used in catalog marketing
Data structure/content known Adds in probability VISUALIZATION – Make data graphical – The issue is most tools are NOT DB-based, rather require data is file-based
Only show “the obvious” Foundation for advanced analytics Sophisticated? You bet – your developers have to know quadratic equations, fourier transforms, etc.


Message to vendors – more intuitive tools for advanced analytics

Visit Amazon or Apple a few times — in truth, get on any site with an online search engine of substance — and you see your preferences and buying behavior have been captured and intelligently mirrored back to you via recommendations. Look to Open Source tools which are emerging as major players in the market.  But while tailored marketing is occurring in the web-based industry, truly pervasive analyses out of the DW which lead to laser-focused cross-sell and up-sell remain elusive. But the information consumers still wait for a small set of skilled data analysts and information / report providers to delve into customer behavior modeling and provide their insights.    Unless you have a set of information providers / analysts versed in the roots of mathematics, don’t expect the tool to do the data mining for you. We still have quite a distance to travel to see all those nifty visualizations techniques like Mind Maps, news aggregators and visualization of connections easily created by the general information consumer which let them mine their data for insights into consumer behavior.