Archive for the ‘Data Quality’ Category
Tuesday, March 13th, 2012
“Mr. Gorbachev, tear down this wall!”
When the Great Communicator made that proclamation, a wall fell. Within months, the world became a different place.
In many organizations, a more resilient wall stands strong. Leveling the divide between business and IT is no mean feat. In fact, technology created and defined that divide. In somewhat of a twist of fate, technology has surprisingly become the enabler, the communicator. Technology available now can help drive collaboration in the business intelligence arena in ways we couldn’t imagine five years ago.
Data profiling is one area where tools get business and IT teams talking. My last blog discussed conducting joint data quality review sessions using data profiling tools as an accelerator. That approach enables a rich conversation between business and IT – a groupthink that creates a high degree of collaboration and trust between the parties that produce and consume data.
Similar collaboration tools are available for monitoring ongoing data quality, maintaining master data, building business glossaries, validating business rules, and other tasks. The tools leverage maturing technology and clever design to enable and enrich the tasks of stewards. These stewardship activities have been difficult to define and harder to implement – until now.
The clarion call here is not about simply deploying new technology – we’ve all been down that road before. Neither do our goals include making every business user a tool jockey. We’ve missed the point if we haven’t grasped the incredible attention to detail required to smooth over the business-IT divide. Tools used with patience and care gracefully handle these kinds of details. Our goal is to obscure the line between technology and business, while eliciting the responsibilities of stewardship.
Using such tools sets the organization into a virtuous cycle of continuous data quality improvement. The business users work with intuitive interfaces that mask the underlying complexities of data, master data and metadata. The tools simplify presentation of information, evaluation of options, and acceptance of inputs. IT gets the feedback needed from the business to improve data knowledge and quality. Players on both sides of the wall gain a deeper appreciation for the need to communicate effectively, while managing data as a corporate asset.
We can turn our organizations into entirely different places by leveraging these capabilities to systematically chip away the divide between business and IT.
This wall, too, is destined to fall.
Monday, February 20th, 2012
Some companies limit data profiling to a tsunami of SQL queries by analysts. This ‘non-scalable’ approach consumes a lot of time and is a tedious and uninspiring activity for a skilled analyst. Most important, this approach does not enable the groupthink of data profiling reviews. For that, we need an accelerator – and a quorum of in-house experts.
Vendors have come to market with toolsets that make it virtually inexcusable to run massive manual SQL checks to profile data. These data quality analysis accelerators provide an effortless and consistent set of data heuristics at the click of a mouse. The tools offer an ad hoc capability to see the data that is both broad and deep. Here are just a few of the items that can be validated: unique values, domain and range of values, default values, data types, field formats, outliers, codeset validity, presence of nulls, blank data, and invalid characters.
The data profiling tools are the accelerator, but the real value comes from the meeting of minds at data profiling review sessions.
Data profiling review sessions need a quorum of business and IT participants. The analyst(s) who wrote the source target mappings and data requirements needs to attend. Invite business SMEs that know and use the data regularly. A QA representative should attend to clarify issues, log the issues into a tracker or enterprise quality tool, and track issue resolution over the coming days.
Sessions are generally guided by the analyst. The source target mappings and business requirements documents should be close at hand during the session for reference. The most useful review sessions have live connections to the data profiling tool and to the data sources. Questions that pop up during the review sessions can be addressed in more detail by drilling into the data profiler, and/or by querying the source data.
Send out the link to the data quality profiling results at least a day before the review session so that everyone has a chance to do reasonability checks. Distribute a checklist of generic data quality pointers. The checklist will direct attention to key fields like primary keys and fields required by the business reports. This will also help ensure a consistent approach to the effort.
Witnessing the groupthink in these sessions is fascinating. Each participant comes to the meeting with a unique point of view on the project inputs and outputs. The data profiling results are parsed to answer questions, generating new questions. A quick exploration back into the source system provides some immediate answers. The data mappings and business requirements are validated. New business rules are proposed on the spot to solve observed issues. The roundtable discussion can be rich and fruitful. This is one forum where the whole is certainly greater than the sum of the parts.
Automated data profiling has become so effortless that we need to consider checking data quality at many points in the development lifecycle – during source analysis, when source data is landed, and after business rules are applied. The data profiling exercise need not be limited to single files or tables. Many profiling tools have inter-table profiling capabilities that can help validate referential integrity and find orphan keys.
Analysts should be analyzing data profiling output, not writing and running endless queries. The delivery team reaches a new level of collaboration when the tools and processes to enable data profiling are part of the team mindset.
Monday, November 28th, 2011
My wife and I are avid fans of the reality show ‘Kitchen Nightmares.’ In case you haven’t seen it, the show’s intimidating host, Chef Gordon Ramsay, is called in to help restore the image and operations of failing restaurants. Ramsay’s formulaic approach is to rework the menu and atmosphere, ignite the enthusiasm of kitchen and wait staff, restore the faith of owners, and re-launch the restaurant – all in the course of a few intense days.
Having seen so many episodes of the show, my wife and I look on in disbelief as the distraught owner goes through yet another exercise in cleaning up the kitchen, tuning the menu, building up team spirit, and bringing the restaurant décor and operations up to 21st century standards. At this point, isn’t the scope of the upcoming effort apparent the minute Chef Ramsay enters the front door?
The Chef points out the subtle and the substantial to struggling owners and workers blinded by years of doing the same thing over and over again.
We in the BI world could use a Chef Ramsay – critic, comforter, and confessor – an oracle and master of all things data warehousing. Think of the messaging the Chef might bring to our workplaces. After years of operating the same way, are we inured to problems and opportunities? What are our resident experts missing that is simply inexcusable? Are the issues painfully obvious?
Imagine the Chef as he steps into your data warehousing shop. Would he advise you to clean up your data quality? Would he admire or be disturbed by your service levels? Would he be proud to say that you deliver ‘the most amazing’ reports capable of modern tools? Would he provide counseling to help rebuild team spirit? Would he shake up the management team? Would he advise that your data warehouse needs a complete makeover and re-launch?
Chef Ramsay achieves the miraculous in just a couple days. It would be unrealistic to expect our data warehouse issues to be solved as quickly, but the Chef offers a fresh perspective that can illuminate our biases.
Thursday, September 22nd, 2011
I was reading through the the 42 CFR 425 Medicare Program: Medicare Shared Savings Program: Accountable Care Organizations yesterday in response to a client asking us about our Managed Services. Specifically thinking about II(C)(2) and data sharing.
The rule discusses data sharing between ACO participants and the government. Generally, the gist of the following few sections suggests allowing the Secretary to share claims information on Medicare beneficiaries both at the point of enrollment as well as on an ongoing basis. Why?
First, having claims data at the point of enrollment helps ACOs satisfy other parts of the rule around creating personalized care plans, helping with care coordination and other changes in care processes that need to be made for that beneficiary–all focused on patient centered principles. The data could be personally identifiable of course.
Finally, on an ongoing basis, the data would be provided in detailed as well as aggregated form to help ACOs understand the total universe of care. Since an ACO may not actually provide all care services for a beneficiary (although the goal is to do as much as possible), obtaining beneficiary level claims data helps ACOs understand the complete picture. The rule goes on to describe how this data would be used in aggregate and in detail (Section 5 and 6):
- Financial peformance modeling
- Utilization management
- Clinical management
- Quality reporting
- Care management/Care coordination
For example, if the ACO’s beneficiary population has a high rate of readmission, then a program could be put in place to improve discharge coordination to reduce readmissions.
Looking over the intended uses, none of which are new per se, it struck me that a lot of Payer claims data analytics is now transitioning to the Provider community. Since ACOs are inherently about Providers, not Payers (which is the primary reason an ACO is not an HMO), the ACOs now need to do Payer analytics.
That’s what Ajilitee is really good at. And we can do this on a Managed Analytics basis so we can help ACOs rapidly get to market on ACO analytics at the beneficiary level. Most importantly, since we build our analytics environments on could computing platforms like Amazon, you only pay for what you need and it can grow to the size you want as the ACO grows.
That’s what struck me about Managed Analytics and the client conversation. With Ajilitee, Managed Analytics has a great promise to deliver capabilities that Payers have today cost efficiently to the Provider community immediately tomorrow.
Tuesday, January 11th, 2011
I’ve had a number of clients and colleagues comment how difficult it is to make Data Governance programs achieve the expected traction in the first year. The learning curve is tough for those new to the concept – with data stewards bearing the brunt of the organization’s expectations for “making things happen.” The pundits, industry experts and Internet community have white papers, blogs, and articles a-plenty providing advice and direction on best practices, things to do and things to avoid, but there exists a void in specific, actionable steps to create a strong foundation. Without these steps, our big ideas and enthusiasm fade in the wake of disappointment and frustration. What’s needed is an approach I call “Pragmatic Data Governance.” Let’s keep the grand plan in our long view but let’s start with some good, solid basics we can build on. I believe Pragmatic Data Governance solutions have some of these characteristics:
• Low cost “must have” tools for Data Governance
• Data Governance service level agreements
• Measuring (and telling leadership about) effectiveness of Data Governance
• Certified data (or what exactly is the data you should certify?)
I’d like to explore how these pragmatic approaches to Data Governance serve as the foundational building blocks for a successful Data Governance program. If you’ve walked this path and have some learnings to share, let me hear from you. Meanwhile, stay tuned for posts on my top four actionable steps to Pragmatic Data Governance.
1 Comment
Category Blog, Data Governance, Data Quality, Information Management, Tina McCoppin | Tags: certified data, data governance, data stewardship, Information Management, service level agreements, SLAs,
