Ajilitee

Qualitative Data Governance metrics can be challenging, insightful

Thursday, September 8th, 2011

In my last blog, I discussed the use of Data Governance Metrics to measure program success.  That blog included a list of 17 typical quantitative metrics. This blog will venture down a much fuzzier path, the use of qualitative metrics to measure DG program success.

Quantitative metrics are gathered directly through the observation and measurement of data.  There is a high degree of transparency and a direct correlation between action and outcome with quantitative metrics. Analytical types like me are easily convinced of the value of Data Governance (DG) using quantitative metrics.  Fix the data, increase the quality score.

Qualitative metrics, on the other hand, are not so transparent. Connecting the dots can be challenging, since the points of data capture for qualitative metrics are often two or more degrees of separation from the data.  That is, a DG project may address (and fix) the quality of the data, but the measurement of success is not the data that was remediated, but some other outcome like a lift in compliance, healthier customer satisfaction survey results, an increase in industry standard scorecards, and so forth.

Qualitative metrics fit into a number of general categories, including compliance, industry ratings, customer satisfaction measures, and business opportunity, amongst others. Successful programs identify metrics meaningful to both middle management and executive leadership.

Compliance – Data governance programs are well positioned to support data-related compliance efforts. Data Governance guides the implementation of controls to document, institute and monitor compliance with data-related regulations.  Cross-functional teams established by DG can look for opportunities to drive cost out of compliance efforts. These regulations include Sarbanes-Oxley, Basel I, Basel II, and, HIPAA.  Compliance requires formal business and management processes to govern the impacted data subject areas that the DG Council can help manage.

Industry ratings –Data Governance can help improve industry ratings such as HEDIS and NCQA scores.  HEDIS enables clients to notify members and providers of the need to obtain necessary services through multiple pathways. Proactive identification of HEDIS care gaps can be directed to DG participants for discussion and targeted improvements. NCQA scores depend in large part on the availability and customer satisfaction with clinical services.   The DG team can help address core data that feeds HEDIS and NCQA scores to help drive a lift in overall patient experiences and customer satisfaction scores.

Customer satisfaction levels – Most companies devote huge amounts of resources to track, measure, socialize and lift customer satisfaction levels. Collection of these metrics can be costly and complex, but need to be considered essential for survival. Commonly used sources include classic data like phone surveys, customer comment cards, and focus groups, to more recent entrants like blogs, Facebook and Twitter.  Collection and analysis of these metrics over a period of time will help gather knowledge of exactly how consumers feel about your products and services and can be linked to improvements wrought by DG efforts.

Business Opportunities – The policies and controls implemented by DG help organizations identify direct and indirect business opportunities.  Correct and current data drives significant and direct impact to the business.  Proposals that carry assumptions based on poor data quality will be at high risk of under or over-bidding. Common themes among the external regulations center on the need to manage risk. The risks can be financial misstatements, inadvertent release of sensitive data, or poor data quality required to drive key decision making.

Sample Qualitative Metrics – Some of the qualitative measures are easier to substantiate than others. For example, you may find it challenging to quantify increased collaboration between teams. You may struggle to link new control standards and policies to increases in customer satisfaction scores. Stay the course! Each DG team needs to define and measure some indirect or qualitative metrics that indicate program success. Brainstorm on the categories in the table below as a starting point for discussion.

There is no standard body of qualitative metrics. Each organization creates their own metrics based on needs, culture, industry, data availability, and so forth. Here are some representative metrics I have used in the past to measure qualitative DG program success in the health care industry.

In my next blog, I’ll explore how to design and implement Data Governance Metrics as foundational building blocks for a successful Data Governance program. If you’ve already walked this path and have some learning to share, let me hear from you.

– Jim Van de Water contributed to this blog.

The need for Data Governance Metrics to measure success

Friday, August 12th, 2011

You made a business case for building a Data Governance organization to solve your Business Intelligence challenges, got your funding, and hired a couple of seasoned consultants to help design the governance strategy, shaped a DG council and stewardship organization, and transcribed policies and procedures to put DG in motion.  A couple of years have gone by and you are still facing those key challenges. What happened?

Is your data governance program working?  How do you know? Are the sponsors happy?  Are the analysts working more effectively? How do you know when the effort has paid big dividends? As Business Intelligence practitioners, we attach metrics to all sorts of activities as best practices.  Shouldn’t capture of success metrics at program inception and on an ongoing basis be a best practice for data governance as well?

Many organizations don’t consider the need for metrics to measure the long-term effectiveness of their Data Governance program. Don’t feel badly – if you haven’t quantified DG success, you’re not alone. Companies in all industries struggle to capture relevant and meaningful DG metrics.  Thankfully, there is no shortage of data that can help you to measure the impact and effectiveness of your Data Governance efforts. The key is to use Data Governance Metrics to measure and demonstrate program effectiveness.

What is Data Governance Metrics?

Data Governance Metrics is foundational to measuring the success and effectiveness of a Data Governance program. Consider the set of metrics designed to measure the effectiveness of the Data Governance. Like other business performance measures, metrics should be managed and tracked at all levels.

What defines a good Data Governance Metric?

Prior to defining the DG metrics, you should understand the key characteristics of a reasonable DG metric and then explore how to map those characteristics to the measurable aspects of data governance. Metrics should be specific, measurable, attainable, realistic and timely.  The following list of questions provides some guidance to jump-start the approach:

• Specific – Identify set of specific metrics that will measure the success of the DG program
• Measureable – Clearly defined, simple to understand and easy to measure
• Actionable – easy to capture, realistic and practical, and quantifiable.
• Realistic – Does the metric have business relevance?—i.e. defined within a business context that explains how the metric score correlates to improved business performance
• Timely – A timely goal is intended to establish a sense of urgency and measure over a period of time to analyze the trend.

The acid test for each metric is whether it is clearly defined, capable of measurement, and directly relevant to improving program effectiveness.  The Data Governance Council will be a good judge, and they should review and approve all metrics.  Metrics are created by either DG Councils or data stewards with input from data analysts.

Key Data Governance Metrics

Data Governance metrics should be identified and tracked prior to implementing the Data Governance program to baseline performance. You will want to capture these metrics periodically and store in a table or database to review progress with the metrics over time.

Commonly measured dimensions of Data Governance include completeness, consistency, timeliness, and uniqueness, although the range of possible dimensions is limited only by the ability to provide a method for measurement. Metrics can be composed of directly measured rules or more complex metrics that are defined as weighted averages of collected scores.  Metrics from a business case or ROI for the DG program proposal can provide a starting point.

At a high level there are two broad categories of DG Metrics:

1. Quantitative Metrics – Data centric Metrics that can be measured as hard benefits like savings in manpower or operational cost savings, etc.
2. Qualitative Metrics – Metrics that measure soft benefits like improved customer satisfaction survey results, increased industry standard data quality scores, etc.

Listed below are some sample Quantitative Metrics I have used in the past to measure the program effectiveness for healthcare clients.

In my next blog, I’ll explore indirect metrics as well as how to design and implement Data Governance Metrics as the foundational building blocks for a successful Data Governance program. If you’ve already walked this path and have some learning to share, let me hear from you. Stay tuned.