CONTACT US 224 265 0400 or Email

The Agile Incubator Blog

Qualitative Data Governance metrics can be challenging, insightful

In my last blog, I discussed the use of Data Governance Metrics to measure program success.  That blog included a list of 17 typical quantitative metrics. This blog will venture down a much fuzzier path, the use of qualitative metrics to measure DG program success.

Quantitative metrics are gathered directly through the observation and measurement of data.  There is a high degree of transparency and a direct correlation between action and outcome with quantitative metrics. Analytical types like me are easily convinced of the value of Data Governance (DG) using quantitative metrics.  Fix the data, increase the quality score.

Qualitative metrics, on the other hand, are not so transparent. Connecting the dots can be challenging, since the points of data capture for qualitative metrics are often two or more degrees of separation from the data.  That is, a DG project may address (and fix) the quality of the data, but the measurement of success is not the data that was remediated, but some other outcome like a lift in compliance, healthier customer satisfaction survey results, an increase in industry standard scorecards, and so forth.

Qualitative metrics fit into a number of general categories, including compliance, industry ratings, customer satisfaction measures, and business opportunity, amongst others. Successful programs identify metrics meaningful to both middle management and executive leadership.

Compliance – Data governance programs are well positioned to support data-related compliance efforts. Data Governance guides the implementation of controls to document, institute and monitor compliance with data-related regulations.  Cross-functional teams established by DG can look for opportunities to drive cost out of compliance efforts. These regulations include Sarbanes-OxleyBasel IBasel II, and, HIPAA.  Compliance requires formal business and management processes to govern the impacted data subject areas that the DG Council can help manage.

Industry ratings –Data Governance can help improve industry ratings such as HEDIS and NCQA scores.  HEDIS enables clients to notify members and providers of the need to obtain necessary services through multiple pathways. Proactive identification of HEDIS care gaps can be directed to DG participants for discussion and targeted improvements. NCQA scores depend in large part on the availability and customer satisfaction with clinical services.   The DG team can help address core data that feeds HEDIS and NCQA scores to help drive a lift in overall patient experiences and customer satisfaction scores.

Customer satisfaction levels – Most companies devote huge amounts of resources to track, measure, socialize and lift customer satisfaction levels. Collection of these metrics can be costly and complex, but need to be considered essential for survival. Commonly used sources include classic data like phone surveys, customer comment cards, and focus groups, to more recent entrants like blogs, Facebook and Twitter.  Collection and analysis of these metrics over a period of time will help gather knowledge of exactly how consumers feel about your products and services and can be linked to improvements wrought by DG efforts.

Business Opportunities – The policies and controls implemented by DG help organizations identify direct and indirect business opportunities.  Correct and current data drives significant and direct impact to the business.  Proposals that carry assumptions based on poor data quality will be at high risk of under or over-bidding. Common themes among the external regulations center on the need to manage risk. The risks can be financial misstatements, inadvertent release of sensitive data, or poor data quality required to drive key decision making.

Sample Qualitative Metrics – Some of the qualitative measures are easier to substantiate than others. For example, you may find it challenging to quantify increased collaboration between teams. You may struggle to link new control standards and policies to increases in customer satisfaction scores. Stay the course! Each DG team needs to define and measure some indirect or qualitative metrics that indicate program success. Brainstorm on the categories in the table below as a starting point for discussion.

There is no standard body of qualitative metrics. Each organization creates their own metrics based on needs, culture, industry, data availability, and so forth. Here are some representative metrics I have used in the past to measure qualitative DG program success in the health care industry.

Category Metrics
  1. Percent of users with access to the PHI data. Access to PHI must be restricted to only those employees who have a need for it to complete their job function.
  2. Number of times data within business critical systems changed or erased  in an unauthorized manner
  3. Decrease in risk or cost of regulatory fines
  4. Decrease in latency associated with delivery of compliance data
Industry Ratings
  1. Contributions to improvement in NCQA report cards
  2. Improved capture ability of HEDIS
Customer Satisfaction Measures
  1. Survey results showing greater collaboration between internal departments
  2. Percent  increase or decrease in customer satisfaction survey index
Collaboration / Improved Productivity
  1. Percent of times DG council detected and eliminated redundant intra- or inter-departmental projects / initiatives
  2. Number of projects that adopted the enterprise logical data model without creating one from scratch
  3. Number of redundant systems eliminated to create a single definition of customer, product, or other widely shared master data

Opportunity / Risk

  1. Business opportunities gained due to better data quality
  2. Business opportunities lost or misaligned due to questionable data quality
  3. Increase in precision of analytics and forecasting gains from improved data quality
  4. Increase in competitive analytics due to data availability and data quality improvements

In my next blog, I’ll explore how to design and implement Data Governance Metrics as foundational building blocks for a successful Data Governance program. If you’ve already walked this path and have some learning to share, let me hear from you.

— Jim Van de Water contributed to this blog.