Ajilitee

5 Key Data Governance Council Metrics

Monday, November 14th, 2011

Ajilitee’s newly-released “Ultimate Guide to Data Governance Metrics for Health Payers” dives into 30+ data governance metrics from a quantitative and qualitative vantage point.  But today, I’m exploring a subset topic, which is how to measure the success of a Data Governance Council initiative.

We’ve said this often, but it bears repeating: data governance programs tend to fall short of expectations because they wind up as tactical data quality initiatives that address accuracy and consistency in silos. They also lack an effective governing body to manage data ownership, lineage and accountability across the enterprise.

We believe that establishing a Data Governance Council is the key to transforming a data governance program into real business value.  An informed and active Data Governance Council will tackle inaccurate, inconsistent and incomplete data holistically through policies and cultural change at the leadership level.  And just as a data governance program establishes metrics on its data quality and performance measurements on the data stewards, it’s equally as important to set performance goals for the Data Governance Council.

In the Health Payer space, metrics are driven by corporate drivers and key performance indicators. Typical drivers include the following:

• Cost avoidance and cost containment
• HIPAA, Privacy and/or regulatory compliance
• Fraud detection
• Constraints management
• Products and Plans time-to-market

A Data Governance Council composed of upper management will be engaged and committed if they are presented with demonstrated successes that address these drivers. This includes continuous data quality that the governance program helps ensure is embedded upstream rather than performed sporadically downstream. Also realized are the benefits of improved transparency, audit-ability and data lineage, which are essential to compliance with government regulations such as HIPAA.

During its instantiation, the Data Governance Council should be reminded that they are part of this group to serve as proactive change agents.  Therefore, this ability to be change agents should be measured. To that end, we recommend these five key metrics to measure the Data Governance Council and its members:

• METRIC 1: Advocacy success measure.
  • Getting each Council member to recognize that their role is not a passive one.   To remain on the Council, they are expected to be “data integrity proselytizers” – e.g., identifying a steward for their line of business, and speaking at their team meetings about the new policies, progress and changes, and so forth.

• METRIC 2: Meeting success measure.
  • Demonstration of commitment. This can be accomplished by an early vote to have a policy that a Council member could and would be “disinvited” for lack of attendance.

• METRIC 3: Each Council Member must bring a Data or Process Issue request to the Council.
  • Demonstration that the Council member understands what is an appropriate process and/or data issue that warrants attention from the DG Council.  They must be willing to push skeletons in their own business areas in front of their peers for resolution.

• METRIC 4: Number of Policies Established.
  • Enterprise Policies serve as the basis for prying systemic data issues away from the silo-minded lines of business. In the first years, typical policies include defining the list of governed data elements; approving Unique Identifier data elements (e.g., Unique Provider, Unique Institution, Unique Member); establishing USPS Address Standardization; conforming Provider Specialty Taxonomy to CMS labels.

• METRIC 5: Maturity Model measure.
  • The Data Governance Council should demonstrate proficiency in their role before tackling the more complex topic of a Data Governance 5-Year Maturity Model.  But by the end of Year 1, the level of progress on the Maturity Model should be set and tracked for each succeeding year.

At each Council meeting, we advocate reviewing each of these Scorecard metrics. Everyone sees the contributions of their colleagues. It’s important to have this level of visibility and openness – peer pressure works wonders! And we stress not to be “locked-in” to a particular set of members. It’s not uncommon to realize that another representative needs to be added or someone cannot make the necessary commitment and should be replaced.

Finally, measure the business impact of a Data Governance Council and then publish results on an enterprise data governance internal website or Sharepoint. This demonstrates the commitment to improved data integrity at the highest executive ranks.

Pragmatic Data Governance—Let’s Get Real, People

Tuesday, January 11th, 2011

I’ve had a number of clients and colleagues comment how difficult it is to make Data Governance programs achieve the expected traction in the first year. The learning curve is tough for those new to the concept – with data stewards bearing the brunt of the organization’s expectations for “making things happen.” The pundits, industry experts and Internet community have white papers, blogs, and articles a-plenty providing advice and direction on best practices, things to do and things to avoid, but there exists a void in specific, actionable steps to create a strong foundation. Without these steps, our big ideas and enthusiasm fade in the wake of disappointment and frustration. What’s needed is an approach I call “Pragmatic Data Governance.” Let’s keep the grand plan in our long view but let’s start with some good, solid basics we can build on. I believe Pragmatic Data Governance solutions have some of these characteristics:

• Low cost “must have” tools for Data Governance
• Data Governance service level agreements
• Measuring (and telling leadership about) effectiveness of Data Governance
• Certified data (or what exactly is the data you should certify?)

I’d like to explore how these pragmatic approaches to Data Governance serve as the foundational building blocks for a successful Data Governance program. If you’ve walked this path and have some learnings to share, let me hear from you. Meanwhile, stay tuned for posts on my top four actionable steps to Pragmatic Data Governance.

Competing for Data Quality

Thursday, September 2nd, 2010

Show them the money! How better data quality lets everyone win.

How do you like this for an IT mantra:

• Inter-company combativeness as a good thing?
• Development vying with QA as a means for cost avoidance?
• Literally, paying for your mistakes?

In the right context and used appropriately, yes – you should not only condone but even actively encourage the “mano-a-mano” competition. Here’s how I have seen it work:

I was part of a relatively young, five-year-old software company at the time – full of enthusiastic, innovative and imaginative folks. The development team was nearing completion of a major release of the product, with significant new functionality and dramatic overhauling of existing logic. Now, some software companies have been known to take a “If it compiles, ship it” mentality. That is, the testing phase is cursory, with the attitude that customers will tolerate and “participate in uncovering” defects and deficiencies. But our CEO made it clear from the outset: We had no intention of making our clients act as our Quality Assurance (QA) function.

But how could we increase the level of product quality?

With a twinkle in his eye, the Director of IT posted a challenge to the ENTIRE company. He said he had established a considerable pool of cold, hard cash – special bonus money for the development team. I’m talking five figures. He told the development team it was theirs to lose. He gave them three weeks to hammer out as many bugs, defects, and issues on the product and supporting documentation as they could. At the end of the three weeks he would have the product and documentation opened up to the entire company for one week. If a non-development individual found a defect, they were paid accordingly:

• $1,000 for Severity 1 (Critical product failure)
• $250 for Severity 2 (High severity)
• $25 for Severity 3+

So, for each issue found post-development, money came out of the development team’s pockets.

I can assure you, those final three weeks of development the lights and laptops were on 24×7. It wasn’t just the money (well, yes, money served as a big motivator). It was also the spirit of friendly competition. Heck, not just your peers, but also the HR Department and the folks in the cafeteria were going after your pot of gold.

And when they got the chance, teams outside of development logged into the testing site and hammered away. Sales and Consulting – the folks on the front lines — went for the big dollars and tried to find the critical breaks. The Helpdesk team ran typical scenarios they encountered when on the phone with customers. The left-brain folks in Accounting and Legal gave long, hard looks at the documentation, on the lookout for discrepancies between what was written and how the product worked.

Everyone was attuned to product quality. And with this technique, everyone could participate in improving the company’s key asset.

The end result? The development team was more thorough in their testing. The “open enrollment” testing period still yielded all levels of severities. And the special bonus was doled out to the deserving participants.

But the big winners were the company’s clients – and thus, in the long run, the company, because the delivered product was more fundamentally sound and bullet-proofed.

How do you calculate the amount of the special bonus pool?

In our case, the Director of IT had estimated the cost of fixing defects post-General Availability (GA) of the product, based on previous releases. Anything less than that is money well-spent. He only instituted this one time, but the lesson was one none of us ever forgot.

What is the cost of fixing defects after release? Google or Bing that question and you find, among other articles:

• Johanna Rothman’s scenarios using this link are from 2000 – and the cost of labor has gone up since that time
• In another article in StickyMinds.com, Rothman proposes the following formula:

• Jon Strickler depicts a chart that shows cost per defect by development phase is
  • Requirements = $139
  • Design = $455
  • Coding = $977
  • Testing = $7,136
  • Maintenance = $14,103
  • OUCH!
• And finally, Capers Jones argues from the vantage point of value of quality based on Function Point analysis.

There’s no argument that identifying and fixing defects earlier rather than later in the development life cycle is less costly – not only in terms of direct cost, but potentially also in the cost of reputation (think of Toyota’s cost to pocket and brand for automotive recalls).

Friendly competition can serve as one avenue for improving a quality release. And save you money in the long run.

Doomed From the Start: Data Warehousing Risks

Friday, August 20th, 2010

Every Project Manager (PM) worth his/her salt (and who is acting in accordance with The Project Management Institute’s guidelines outlined in the “Practice Standard Project Risk Management” manual) has a list of all the things that can go wrong during a project, as captured in the Project Risk Log. The good PMs work with the project leadership (the business sponsor, the IT executive, team leads, analysts) in a Risk / Mitigation session at project commencement—anticipating any potential chinks in the project’s armor. And they hold a weekly meeting to review and update the risks for the project’s duration. 

So, having seen how risks become issues and issues become scope change / budget increase / missed project deadlines, I had one savvy client ask as part of their data warehouse request for proposal (RFP) process that each service provider bidding compile a list of their top five risk factors and mitigation steps in order to achieve a greater degree of confidence that the project would be delivered within the budget. The list that emerged consisted of the following:

1. Consensus on and completeness of reference data and hierarchies with an ability to reconcile across business units and functions
2. Scope creep and SME expectation
3. Availability of SMEs, system experts and other pertinent resources, timeliness of decision-making
4. Excessive complexity of reporting environment and / or data transformation
5. Lack of data cleanliness

Before I go any further, I’ll concede that this list is not comprehensive. And I’ll also acknowledge that risks need to be assessed for each project. But I stand by my assertion that this set of risks creates huge headaches if they are not addressed and tackled head-on as early as possible.  Let’s explore further.

Risk #1: Reconciliation of Reference Data and Hierarchies 

Project Status: Stalled. Evidenced usually in Requirements and Analysis stages. Reason: We can’t get the Finance SME and Marketing SME to agree on the Product Hierarchy or perhaps align on the definition of Net Revenue.  If the data warehouse is the central “single version of the truth,” how do we keep moving forward if the LOBs have problems agreeing on that single version?

Mitigation Strategy

1. Clearly defined point of authority
2. Early start of this task
3. Emphasis on financial reconciliation
4. Use of commercial tools (COTS) accompanied by short-term consultation to leverage industry solutions
5. Careful attention to managing the schedule for early identification and impact of prolonged data investigation and data resolution
6. Weekly monitoring of budget using predictive tools

Risk #2: Scope Creep

Project Status: Behind schedule. Evidenced usually in Design and Development stages. Reason: Requirements keep changing. As we moved from the Analysis Phase into Design Phase and your designers had questions about the data or the “meaning” of a requirement, your Business Analyst team would go to the LOB for answers and come back with more requirements.

Mitigation Strategy

1. Careful documentation
2. Review of requirements by the business, as well as intermittent audits and evaluations by PMO Quality Assurance
3. Cross-reference requirements to project plan
4. Manage expectation-setting of SMEs and business units
5. Careful impact analysis and change control
6. Personal accountability of project members
7. Open, honest and ego-free environment

Risk #3: Availability of SMEs

Project Status: Behind schedule. Evidenced usually in Design stage. Reason: Design and development team asked for business rules on how to handle exceptions, or create a metric, etc., but the subject matter expert was on vacation or wouldn’t respond. The design team forges ahead with a “best guess.”

Mitigation Strategy

1. Early notification of SME participation, including expectations of amount of time over duration of project that will be needed and committed to
2. Consideration of SME time – avoid wasting their time through the appropriate skill-set and preparedness of the BA team and facilitators
3. Involvement of the “right resources” rather than a less-informed substitute
4. Effective scribes during interviews and meetings: Note-taking, action list and follow-through

Risk #4: Complexity of the Reporting Environment and Transforms

Project Status: Behind schedule. Evidenced usually in Development stage.  Reason: Underestimation of the number of reports, complexity of the reports, metrics and measurements needed for the reports, performance of the reports.

Mitigation Strategy

1. Early evaluation of complexity of reports and queries
2. Validation and clarification of business requirements, as well as early and careful translation of the data requirements
3. Validation of the number and accessibility of source data fields (get a sense of this before project kick-off)
4. Validation of the number of reports (get a sense of the number of existing reports that will be replaced as well as how many new reports will be created)
5. Project tracking
6. Addressing data problems as they occur
7. Incorporation of checkpoints and team structures. E.g.:
   • Project Management Office
   • Staffing Team
   • BI-COE or BICC
   • Training Office

Risk #5: Data Cleanliness

Project Status: Behind schedule (or worse – on schedule and “discovered” during Testing/QA). Evidenced usually in Development or Testing/QA stages.  We’re always promised at the beginning of the project that the quality of the data is good / acceptable and we’re always so shocked when the development team says they are rejecting a large percent of the data (unless they just pass through everything. Yikes!) Or – WORSE – the  Testing/QA team finds it out. Or – WORST CASE SCENARIO – in production the information consumers start screaming that the data warehouse is unusable.

Mitigation Strategy

1. Formal data sampling and profiling early in the project (get that development environment set up right away, during requirements gathering, and start getting samples from the source systems)
2. Early attention to the definition of data quality criteria (need those SMEs again) — and refer to Jill Dyche, who has continually sounded the clarion call for DQ
3. Early evaluation of complexity (your Architect is key)
4. Validation and clarification of requirements, as well as early and careful translation of the data requirements
5. Project tracking
6. Addressing data problems as they occur
7. Use of commercial tools (COTS) accompanied by short-term consultation
8. Point of authority for quality issues and prioritization 

Implications of Risk Mitigators on Project Delivery

As noted, early and ongoing evaluations of these risks should be conducted. For those risks which are determined to be of a greater degree than an anticipated “acceptable level of risk,” (e.g, excessive degree of report complexity or greater number of data quality issues than anticipated), the joint team of Business and IT project leadership must work to achieve a balance for inclusion / exclusion into the effort while still meeting the project’s objectives.

What are Your Top Project Management Risks?

Perhaps I’ve been “preaching to the choir,” and all you saavy PMs out there already observe these best practices.  On the other hand, we’ve all lived through nightmare projects at some point in our careers and carry with us bitter lessons learned.  What are the top 2-3 project risks that keep you awake at night?  Would you like to share your horror stories and lessons learned?  (Hmmm…no names, please. Use the “I have a friend with a problem…”)

BACK TO ALL BLOG POSTS>>

Are “advanced analytics” really all that advanced?

Saturday, July 10th, 2010

We have roughly 20+ years of BI/DW projects under our belt – but how satisfied are we at the extent or depth of our reporting and analytic progress?…that the promise of advanced analytics is being realized?…and that we are operating as an agile organization, with the ability to derive meaningful insights for decision-making and taking action? Before approaching this topic, it’s useful to distinguish categories of reporting and then think about the Information Providers (those who create the reports) and Information Consumers (users).

Categories of Analytics

1. Standard Retrospective Reporting: At a minimum, an organization tracks key performance indicators, which serve as leading or lagging measures. Typically the information consumer has a sub-set of selected, pre-defined selection criteria, measurements and metrics available which can be combined and segmented.

2. Exploratory Analytics: Being able to infer patterns is a more advanced technique for deriving insight from data. Power analysts/ report developers serve as the “go-to” folk for the business lines (and typically have a never-ending backlog of requests from the information consumers they support).

These first two categories help us identify exceptions and answer basic questions. They explain the past or current…and they are the primary focus of the BI tools and OLAP available in the market.

We’re no longer content with mere “trends” out of our data. Today’s DWs with huge amounts of data can highlight for us where the exceptions and anomalies reside. The TRUE value out of our data requires statistics, data mining, regression analysis, complex models and algorithms.

3. Predictive Analytics: Ah, this is the place…the BI/DW vision being realized.  It’s where we gain an understanding of complex relationships between data; have the ability to chew through mind-boggling amounts of data; and predict behavior of our consumer.  But it remains neither easy nor intuitive.  A select few of our information providers still roll-up their sleeves and go through the rigor of regression analysis, statistics and probabilities. Twenty years and we’re still reliant on understanding advanced math for getting answers?

So while considering predictive or ‘advanced’ analytics, I come across the interesting link http://www.smashingmagazine.com/2007/08/02/data-visualization-modern-approaches/…and am visually awed at the range of presentations of data. YES! –Look at all that data from all these customers across all these years being visually captured and conveyed. But…sadly, improved design in our tools does not mean advanced analytics. To remain competitively viable, we have to be able to deduce important consumer behavior. Querying and reporting do NOT provide this – the volume and complexity of the data means you need more than SQL and your basic BI tool usage.

Advanced Analytics

TDWI lecturer Mark Madsen asserts that advanced analytics is more feasible now because “the drivers behind this are not necessarily new algorithms or better tools, but the availability of computing power and storage at affordable price ranges…caused by the fact that nowadays it is feasible to analyze many gigabytes of data in a near real-time fashion, something that was impossible to do 5 years ago)”. He describes how advanced analytic techniques and tools are out there for all types of data mining usage (and some are free!).  But you have to figure out which tools / techniques (see table below) apply to and are most suitable for your ultimate purpose: Churn/attrition modeling?…customer acquisition?…behavioral advertising?…email targeting?…CLV Modeling?…product recommendation system?…upsell/cross-sell?…click fraud detection?

Message to vendors – more intuitive tools for advanced analytics

Visit Amazon or Apple a few times — in truth, get on any site with an online search engine of substance — and you see your preferences and buying behavior have been captured and intelligently mirrored back to you via recommendations. Look to Open Source tools which are emerging as major players in the market.  But while tailored marketing is occurring in the web-based industry, truly pervasive analyses out of the DW which lead to laser-focused cross-sell and up-sell remain elusive. But the information consumers still wait for a small set of skilled data analysts and information / report providers to delve into customer behavior modeling and provide their insights.    Unless you have a set of information providers / analysts versed in the roots of mathematics, don’t expect the tool to do the data mining for you. We still have quite a distance to travel to see all those nifty visualizations techniques like Mind Maps, news aggregators and visualization of connections easily created by the general information consumer which let them mine their data for insights into consumer behavior.

BACK TO ALL BLOG POSTS