The right combination of proven quantitative and qualitative metrics will help improve your information, and also will drive continued executive support for your program.  Please enjoy our complimentary guide.

DOWNLOAD:  The Ultimate Guide to Data Governance Metrics for Health Payers:  30+ Ways to Discover and Score Success in 2012

Ajilitee is committed to ensuring that your privacy is protected.  Please view our full privacy policy here.

Vendors have come to market with toolsets that make it virtually inexcusable to run massive manual SQL checks to profile data. These data quality analysis accelerators provide an effortless and consistent set of data heuristics at the click of a mouse. The tools offer an ad hoc capability to see the data that is both broad and deep.  Here are just a few of the items that can be validated: unique values, domain and range of values, default values, data types, field formats, outliers, codeset validity, presence of nulls, blank data, and invalid characters.

The data profiling tools are the accelerator, but the real value comes from the meeting of minds at data profiling review sessions.

Data profiling review sessions need a quorum of business and IT participants. The analyst(s) who wrote the source target mappings and data requirements needs to attend. Invite business SMEs that know and use the data regularly. A QA representative should attend to clarify issues, log the issues into a tracker or enterprise quality tool, and track issue resolution over the coming days.

Sessions are generally guided by the analyst. The source target mappings and business requirements documents should be close at hand during the session for reference. The most useful review sessions have live connections to the data profiling tool and to the data sources. Questions that pop up during the review sessions can be addressed in more detail by drilling into the data profiler, and/or by querying the source data.

Send out the link to the data quality profiling results at least a day before the review session so that everyone has a chance to do reasonability checks. Distribute a checklist of generic data quality pointers. The checklist will direct attention to key fields like primary keys and fields required by the business reports.  This will also help ensure a consistent approach to the effort.

Witnessing the groupthink in these sessions is fascinating. Each participant comes to the meeting with a unique point of view on the project inputs and outputs. The data profiling results are parsed to answer questions, generating new questions. A quick exploration back into the source system provides some immediate answers. The data mappings and business requirements are validated.  New business rules are proposed on the spot to solve observed issues. The roundtable discussion can be rich and fruitful. This is one forum where the whole is certainly greater than the sum of the parts.

Automated data profiling has become so effortless that we need to consider checking data quality at many points in the development lifecycle – during source analysis, when source data is landed, and after business rules are applied. The data profiling exercise need not be limited to single files or tables. Many profiling tools have inter-table profiling capabilities that can help validate referential integrity and find orphan keys.

Analysts should be analyzing data profiling output, not writing and running endless queries. The delivery team reaches a new level of collaboration when the tools and processes to enable data profiling are part of the team mindset.

Founded in 2008, LaunchPoint has served 30 customers through its two divisions: Ajilitee, a consulting and services firm that specializes in agile analytics, business intelligence, cloud enablement, and information management; and Discovery Health Partners, a provider of intelligent healthcare cost containment solutions that include subrogation, coordination of benefits and dependent eligibility verification.  Customers include leading healthcare payers, provider networks, academic medical centers, and self-insured corporations.

Read full press release here>>

BI power users build complex reports, drive statistical models, satisfy ever changing regulatory reporting, and drive sophisticated analysis for your management team. These folks do the heavy data lifting at your organization. They are the data gurus, good ones are hard to find, and they don’t work cheaply. Without a sizeable contingent of these brainiacs, you’re stuck in a world of basic operational reporting. With them, you can rule the data that drives your business and sound decision making. You don’t have to wish for power users. Create them. This blog series will explain how.

There are many types of thinkers working in your organization. The people who make up your analytical community are no exception. Brain science splits learners into two camps – the visual and the logical learners, corresponding to the right and left hemispheres of the brain. The way to train your analysts is to cater to how each of their brains operates by bringing a little science into your tool selection, user enablement and training.

What is the right approach? You could survey each analyst to discover their learning style, then craft an optimal learning path with customized toolsets and techniques. That approach is likely to be slow, painful and expensive. Instead, tap into their instincts by leveraging their natural learning style. Latch onto their brains.

What does a brain-centric approach to creating BI power users look like?

As a visual thinker, SQL training threw me into a proverbial rabbit hole. I was set back months in my development as an analyst. Several years later when I started training end users on tool use, I thought everyone wanted to start out building querying skills using visual tools. I was fooled – twice – maybe you have been too. My predilection for visual tools is just as disabling a bias as a focus on pure SQL. We need to work with both camps. Truth be told most of your users do prefer starting with visual tools. On the other hand, some of your users are already comfortable with visual tools and need to work with code to enable more productivity. Forcing a right brainer into code too quickly, or a left brainer into image based analysis, is a mistake that is counterproductive to human growth.

The right mix of tools and support plays to both sides of the brain to enable your users to grow at an optimal pace. Visual enhances code, code explains and reinforces visual. Access to both types of tools, as well as the hybrid visual tools that allow code display – will please all your fledgling brainiacs. The better your development program accommodates their brains, the quicker you’ll see results. The ‘secret sauce’ is recognizing the delicate dance between images and words that deepens understanding. Power users’ brains work with a rich set of visual and coded tools.

I propose that the reason it is so difficult to grow BI power users is that we continue to ignore differences in how people think. We deploy tools using data requirements, not people requirements, and certainly not thinking requirements. Don’t ignore the differences. Grow your analysts incrementally by bringing their visual and logical parts into lockstep.

In my next blog, I’ll trudge deeper into this topic to explore how the right triage of tools and capabilities can help build that elusive BI power user community.

Move the code into the test environment after it has been thoroughly system tested.

I’ve never understood why projects rush to get code into the test environment. Think of the constraints on our working environments. Development is owned and controlled by the developers. Test, on the other hand, is locked down, typically by the QA team (and by all rights should be).  Working through functional canary tests in development is quicker, easier, and cheaper.

Execute the vast majority of system tests on the development integration platform.

A common practice is to migrate work from developer “sandboxes” to a development “integration” area, where code is assembled into a BI solution. The integration area is the ideal test platform. That move is an at least an order of magnitude easier than moving the code out of your freedom loving development environment to the lockdown on test.

Make system testing a formality.

Development of system test materials, scripts, and the test harness needs to coincide with the arrival of unit code into the development integration environment. Require the development team to demonstrate successful system testing within the development integration area BEFORE moving the code into the test area. The goal is to make system testing a formality. (Mindful readers will note that I stated just the opposite in part #5. The rule only applies if you complete the majority of system testing on development).

Regression test by design.

Too many projects focus on passing failed tests while ignoring previously passed tests.  The fallacy is that testing is complete when code fixes pass a retest. The reality is that code fixes can impact code that ran successfully in the past, causing test failure. Every set of code fixes mandates a test rerun.  Fortunately, our automated and fast-running functional canary test datasets means you can afford lots of test cycles (see part #6).

Every incremental code change requires the addition of a small set of tests to the functional canary test dataset. The entire test dataset runs together to verify that the incremental change and all prior code perform as expected. Each functional canary test set run validates the entire code base. This ‘black box’ treatment of the code (see part #2) ensures that every logic path – old and new – is retested each time the code is changed.

This is regression testing by design.

System testing expands to encompass new rules as the code base matures. Our functional canary data sets and scripts expand in scope and precision as the code base matures. The code base and system testing mature in lockstep, all outside the confines of the test platform.

Rerun your test scripts on the system test environment to validate code promotion.

Typical project plans deliver system test scripts and data sets just in time for system testing. Initial migration from development to test uncovers defects related to migration. Errors are found in the setup of system test data sets and test scripts – even if the code is working perfectly. This happens when neither the migration process, the test setup, or the code base are isolated. Discovering the source of issues is going to take time – and yet more migrations. This is an expensive way to validate testing and migration.

Let’s say the code passes all system tests prior to migration to the system test environment. You will know (in development) how well the code is working. The test scripts will be debugged before they reach the test environment. After promotion, repoint the test harness to the new environment and the right datasets and you’re ready to rerun system testing on the test box. The system tests will now focus on the validity of migration and setup of the test environment.  Any differences can be attributed to those issues, not bugs in the code or test scripts. This can reduce from weeks to days the time it takes to validate that the migration process works correctly. This is a cheap and expeditious way to validate migration.

Success is in the timing and coordination.

System testing done this way requires some TLC and more attention from management.  Development of the test scripts and the test harness need to be timed with the development and integration of the solution components. More coordination is required between developers writing code, creating test cases and test data.

Rethinking system testing

The testing process described in the seven parts of this blog may be seem counter-intuitive, given that what is proposed is the completion of most system testing before code touches the test box. You’ll need to rethink how your team conducts testing, to consider the content and timing of test scripts and data sets, to offer guidance to the development and testing teams, and to find a skilled developer to tie the tests and validations together with appropriate automation (see part #6). None of these are beyond the skills of an experienced development team working with a good project manager.

This approach to system testing can help hit project objectives and deliver the project sooner, to the kudos of management and the end user community.

Good luck with your system testing efforts.

Please share your thoughts about this blog, or relate your own experiences.

- Jim Van de Water contributed to this blog.

Having seen so many episodes of the show, my wife and I look on in disbelief as the distraught owner goes through yet another exercise in cleaning up the kitchen, tuning the menu, building up team spirit, and bringing the restaurant décor and operations up to 21^st century standards.  At this point, isn’t the scope of the upcoming effort apparent the minute Chef Ramsay enters the front door?

The Chef points out the subtle and the substantial to struggling owners and workers blinded by years of doing the same thing over and over again.

We in the BI world could use a Chef Ramsay – critic, comforter, and confessor – an oracle and master of all things data warehousing. Think of the messaging the Chef might bring to our workplaces. After years of operating the same way, are we inured to problems and opportunities? What are our resident experts missing that is simply inexcusable? Are the issues painfully obvious?

Imagine the Chef as he steps into your data warehousing shop. Would he advise you to clean up your data quality? Would he admire or be disturbed by your service levels?  Would he be proud to say that you deliver ‘the most amazing’ reports capable of modern tools? Would he provide counseling to help rebuild team spirit? Would he shake up the management team? Would he advise that your data warehouse needs a complete makeover and re-launch?

Chef Ramsay achieves the miraculous in just a couple days. It would be unrealistic to expect our data warehouse issues to be solved as quickly, but the Chef offers a fresh perspective that can illuminate our biases.

The right combination of proven quantitative and qualitative metrics will help improve your information, and also will drive continued executive support for your program.  Please enjoy our complimentary guide.

DOWNLOAD:  The Ultimate Guide to Data Governance Metrics for Health Payers:  30+ Ways to Discover and Score Success in 2012

Ajilitee is committed to ensuring that your privacy is protected.  Please view our full privacy policy here.

We’ve said this often, but it bears repeating: data governance programs tend to fall short of expectations because they wind up as tactical data quality initiatives that address accuracy and consistency in silos. They also lack an effective governing body to manage data ownership, lineage and accountability across the enterprise.

We believe that establishing a Data Governance Council is the key to transforming a data governance program into real business value.  An informed and active Data Governance Council will tackle inaccurate, inconsistent and incomplete data holistically through policies and cultural change at the leadership level.  And just as a data governance program establishes metrics on its data quality and performance measurements on the data stewards, it’s equally as important to set performance goals for the Data Governance Council.

In the Health Payer space, metrics are driven by corporate drivers and key performance indicators. Typical drivers include the following:

• Cost avoidance and cost containment
• HIPAA, Privacy and/or regulatory compliance
• Fraud detection
• Constraints management
• Products and Plans time-to-market

A Data Governance Council composed of upper management will be engaged and committed if they are presented with demonstrated successes that address these drivers. This includes continuous data quality that the governance program helps ensure is embedded upstream rather than performed sporadically downstream. Also realized are the benefits of improved transparency, audit-ability and data lineage, which are essential to compliance with government regulations such as HIPAA.

During its instantiation, the Data Governance Council should be reminded that they are part of this group to serve as proactive change agents.  Therefore, this ability to be change agents should be measured. To that end, we recommend these five key metrics to measure the Data Governance Council and its members:

• METRIC 1: Advocacy success measure.
  • Getting each Council member to recognize that their role is not a passive one.   To remain on the Council, they are expected to be “data integrity proselytizers” – e.g., identifying a steward for their line of business, and speaking at their team meetings about the new policies, progress and changes, and so forth.

• METRIC 2: Meeting success measure.
  • Demonstration of commitment. This can be accomplished by an early vote to have a policy that a Council member could and would be “disinvited” for lack of attendance.

• METRIC 3: Each Council Member must bring a Data or Process Issue request to the Council.
  • Demonstration that the Council member understands what is an appropriate process and/or data issue that warrants attention from the DG Council.  They must be willing to push skeletons in their own business areas in front of their peers for resolution.

• METRIC 4: Number of Policies Established.
  • Enterprise Policies serve as the basis for prying systemic data issues away from the silo-minded lines of business. In the first years, typical policies include defining the list of governed data elements; approving Unique Identifier data elements (e.g., Unique Provider, Unique Institution, Unique Member); establishing USPS Address Standardization; conforming Provider Specialty Taxonomy to CMS labels.

• METRIC 5: Maturity Model measure.
  • The Data Governance Council should demonstrate proficiency in their role before tackling the more complex topic of a Data Governance 5-Year Maturity Model.  But by the end of Year 1, the level of progress on the Maturity Model should be set and tracked for each succeeding year.

At each Council meeting, we advocate reviewing each of these Scorecard metrics. Everyone sees the contributions of their colleagues. It’s important to have this level of visibility and openness – peer pressure works wonders! And we stress not to be “locked-in” to a particular set of members. It’s not uncommon to realize that another representative needs to be added or someone cannot make the necessary commitment and should be replaced.

Finally, measure the business impact of a Data Governance Council and then publish results on an enterprise data governance internal website or Sharepoint. This demonstrates the commitment to improved data integrity at the highest executive ranks.

In other words, here comes the painful part. Well, then again, maybe not.

Here is what you might expect to see for a well-vetted solution:

• 20 to 100 system test executions during development
• 3-5 system test executions during testing
• 2-3 system test executions during deployment to production

Let’s rephrase it; a BI project should plan for 25 to 100+ system test cycles.

I know what you’re thinking – that a LOT of tests.

This is not only possible, it will also help ensure your longevity at the company. You need to design system testing to be executed and evaluated quickly – in fact, very quickly. That’s why functional canary testing is so vital – it’s quick and thorough and it needs to occur in every system test cycle. That’s also why we’re going to build automation into our system testing routine.

Test counts might look something like this:

• 100 Functional canary
• 20   Large data sets
• 60   Incremental
  • • 50 Incremental functional canary
    • 10 Incremental large data

You may have also have noticed that the system testing is heavily skewed into development.  Is this your concept of system testing? Read on.

Most system testing cycles should occur while the code is in the development environment. System testing executed in the development environment mitigates for the conditions we discussed in the last blog, “The Case for Thorough System Testing”. When system testing occurs after development, your code will be subject to a vicious cycle of unexpected new rules, recoding, and retesting.

The concept of system testing is often tightly coupled with Quality Assurance. QA generally starts their efforts when development is completed. Old school! Your project must find a way to have QA teams develop and/or approve system test data sets and methods to execute during development. This is a critical concept that project leadership typically struggles with or simply does not understand.

Target functional canary test data set runs to complete in thirty minutes or less. Test cycles include setup, execution, and validation of test results. The big challenge is to complete the result validations in that timeframe. Traditional system tests are time consuming as they require humans to compare results against expectations. Manual test validations can take upwards of two days to complete. Sorry, that simply won’t meet our timeline when we have dozens of tests to execute. Efficient and fast system testing and validation require automation.

One test concept automates system test setup and cleanup using a “test harness.”  A test harness includes logic that differentiates between all of our test conditions – functional canary data set and large data set runs, as well as initial and incremental data set runs.  A good test harness provides the opportunity to the tester to select the type of test run and the locations of source and target data.

Test validation is done using a preconfigured setup of test target data sets. This could be actual data processing results (e.g. a data mart schema) or a test view generated on top of the actual data processing results (e.g. summary derived calculations from the data mart). A data set defines the test, the table/view, row/column, expected value, when the test executes, the test run, actual data values and test results. Manual reads are replaced with an algorithm that reads and writes to the above dataset, finds the comparison value, records the test cycle and results, and calls out variances. It’s an elegant system that captures all the required metadata and results for our developers to analyze just the unexpected values.

Your team has a lot of work ahead planning and executing a series of system test cycles in development, QA and production. How do you plan to get all of this done in a reasonable timeframe?

My next blog will discuss additional guidelines for successful system test execution.

- Jim Van de Water contributed to this blog.

Imagine if you will a crystal ball on the desk in front of you.

Inside the orb, you can view your project performance.  You see a project that gathers all the business rules correctly and understands and mitigates data quality issues. You see developers implement complex rules and properly integrate and test their code. At completion, a satisfied business user community praises the BI team.

Unlike the crystal ball, where all the business rules are known and data quality is predictable and resolved flawlessly, reality can be opaque and imperfect.  System testing can help us address both the substantial and subtle obstacles.

System testing highlights shortcomings in our business rules and brings unforeseen data quality issues to light.

The validity and scope of the business rules defined for a BI effort may not be apparent until system testing starts. This maxim doesn’t prevent our diligent business analysts and development teams to offer up their best attempts to fill out incomplete, vague, or poorly written business rules.   Challenges with data quality can appear anywhere. Unit testing rarely exposes such problems. Neither does cursory system testing. No business analyst or user can anticipate all possible data conditions, so the rules defined to address data quality will come up incomplete at best, or even incorrect.

System testing authenticates our designs.

Complexity motivates our designers to break solutions apart. Similar logic is consolidated rather than distributed for the sake of consistency and maintenance.  Performance considerations mandate certain expensive logic be performed first. The design solution was parsed out amongst the development team. Our competent developers completed and unit tested their code components, but assembly concealed unexpected ‘system’ behaviors. Unit testing does not address these anomalies.  The interplay of independently developed code sets within a designed solution can cause our work to miss the mark. The business goals can become anything but transparent in the designed solution.

Forget the idea that system testing is just a formal validation of development. It’s not. Done right, system testing will validate your business rules, elicit acceptable data quality, and prove out the solution design. All those steps are essential if you want to deliver a trustworthy solution to your end users.

What about that crystal ball sitting on your desk?

Use it to magnify, not mask, project issues.

My next blog will explain how to automate your system data sets.

- Jim Van de Water contributed to this blog.