Many healthcare insurance companies are not known for rapid innovation. But times have changed. The need to manage members using Customer Relationship Management (CRM) technologies has greatly increased and many Payers have started their CRM efforts. But, these CRM technologies have been in use other industries for 20 years.  The technologies needed to support CRM analytics and member point of care/point of service interactions are different then what many Payer organizations have in place today.

Where do bigdata technologies play? We have been asked that question here at Ajilitee and as part of working on our FWA products and internal R&D innovation, we have been using and looking at these technologies for awhile.

Bigdata technologies span many areas and have interesting names: hadoop, hbase, hive, pig, nosql, mahout and more. Nosql and hadoop in particular are core technologies that other layers build on. For example, hive and pig build on hadoop. Hadoop itself can build on nosql technologies. I will not repeat all of these concepts in this blog because a lot of content has already been written on these technologies. From here on, I’ll assume you are familiar with some of these terms or look them up easily.

Guiding Thoughts

Here’s a couple of principles that can help guide the conversation:

• Deep insights: The more you understand your domain specific problem, the easier it is to adapt new technologies to solve them in novel ways or to recognize trade-offs you are implicitly making with current technologies. You can find novel uses that solve a problem in different ways. Think Geoffrey Moore and disruption.
• Delamination: By rethinking the layers of technologies you use today, its possible to pick and choose the layers and how they combine together to create new solutions. Many nosql packages do not include compute engines within their data access and storage solution. So solutions such as hadoop must be layered on top of the nosql databases in order to obtain compute processing. There are a lot of variations to this thought but just think “delamination” can be helpful around innovation.
• Do something different: Unless you are really, really smart, a great way to learn about how to use the new technologies is try something and fail multiple times to solve a problem.  This learning by doing is key to innovation. Of course, you have to learn from your mistakes each time—failing by itself is pointless.

Scanning the bigdata Technical Landscape

There are many new technologies in the bigdata world. Let’s look at nosql. Many people question whether the technologies are mature enough for production use and whether they help them solve business problems faster, cheaper or better in some way.

Bigdata and nosql conversations usually start with explaining issues found in the world of managing and serving massive amounts of data needed for websites. But the technologies seem confusing at times and begin to wonder if they apply to our problems especially those in the healthcare world.

Nosql technologies are often quoted as having the following properties:

• No sql: This means there is no sql!
• Schema-less: There is no schema!
• Eventually consistent: Forget ACID. Forget transactions. Eventually your data is consistent.
• Fault tolerant, scalable, distributed: A whole number of really great architectural characteristics that sound good but you are not always sure apply to you.

Once you start looking at the bigdata technologies you are immediately struck by the fact that:

• With some nosql technologies you actually do define a schema and indicate what a column’s type and name is. Some nosql technologies also need to know how to sort columns so they need knowledge of how to compare keys or values. That sounds like a schema!
• With nearly all bigdata technologies, someone has already written a [insert technology name here] Query Language to you can run queries. This sounds like everyone still wants *SQL.
• With some nosql technologies, you have to specify properties such as coherency, which indicates that you can get the value you just wrote back out of the database. While not the full definition of ACID, it starts sounding close!
• There is a lot of parallelism everywhere. The file system is parallelized! Don’t forget that the job flow is parallelized as well! (assuming it fits a specific data parallel processing model). Everything is about scale-out—add more nodes and everything keeps running, the system stays up, and everyone has fast, guaranteed access!
• There appears to be several interfaces into the database some of which require programming in a language you may not be familiar with e.g. not SQL! How do you even load data?! You almost feel like you are programming at the lowest level of database programming possible. Wait a second, which layer are you programming? The filesystem level? The map-reduce level? Or both?

This seems very confusing. So let’s think through the issues. We want to avoid having everything look like a nail because we have a bigdata hammer.

Healthcare Example

In the Payer space, Master Data Management (MDM) is finally becoming a component of the business and architectural landscape.  In the Payer world, MDM means managing Providers, Members, Contracts and Products and other business entities that you can often touch and feel or that are really considered non-claims data.  Other types of Payer data include “event” data. This is data generated by interactions with Members from sales, service and care oriented interactions. Of course, there is also claims data. Claims data is the largest source of data, followed by “event” data then MDM. Are some of the bigdata technologies relevant for even the small datasets such as MDM datasets? Small in this case means a few million rows and typically much less. Of course, this links back to our FWA problem statement at the beginning of the blog and the need to clean up our Provider list to perform Fraud analysis. We’ll illustrate some of our points with industry specifics.

• Schema-less:
  • The NPEES Provider list from CMS changes multiple times a year. The list contains all of the Medicare Providers and some, but not all, of their demographics.  The columns of data change although not frequently. New providers are added or removed as appropriate. Provider data inside a Payer typically originates in several systems–sometimes up to 20.  So it would seem that a technology that claims to be “schema-less” would be useful. But schema-less does not mean that you do not need to specify the data types of the data.  You have to specify the format somewhere so external tools can use the data. The NPEES file has several sub-entities in it like addresses and other codes indicating the Provider’s specialty or whether the Provider is an individual or an organization. Shouldn’t we pull these sub-entities out and make them their own table? Shouldn’t we also try to specify where and how the data should be loaded to be efficiently accessed, perhaps by using table partitions, or striped volumes or other typical database designs? These are normal database design concepts.
    • Part of the value of being schemaless is that you tend to concentrate data together into denormalized structures and use it to answer a smaller set of business questions such as “what data changed between NPPES files each month?” And the data you load may be very dirty, so lets load it all as strings, then convert the data in the database itself. We don’t have to work to hard to specify types, but we must specify some. We can also ignore doing detailed table design because most nosql database are designed to scale out. Bigdata can help us push aside this operational complexity.
    • MDM data changes over time. For example, you may choose to append one external data vendor’s Provider demographic data one year, then switch the vendor the next. That’s a whole new set of data structures in the traditional world. Being schemaless allows us to manage data changing over time without having to reload or re-baseline to achieve acceptable performance. Hence, you can evolve the schema more easily and that’s a great reduction in operational complexity.
    • No-sql: We clearly need to write a query to determine what changed between different NPPES file releases. We have to write a query. The entire claim of nosql must be false! The answer is more subtle than that. The claim of nosql is really one of not having many characteristics of traditional RDBMS databases built into the database layer. For example, you will not see nosql databases implementing referential integrity through sql statement such as foreign keys, etc. You do have to specify primary keys for some nosql database just to help with managing the data.  In fact, many systems today, whether a data warehouse or a transactional system, actually implement integrity in the processing layer above the database these days. This is neither wrong nor right, but just where it is often happening. Hence by saying its nosql, you are really saying that the data architecture is one where the data is more concentrated, where integrity is implemented in a processing layer and not the database, and where the data access interface makes as few assumptions about the data as possible in terms of its structure.
      • There is often another implication of schemaless that is less often recognized. Because the nosql database essentially delaminate the database stack to some degree. Mathematical processing occurs outside the layer.. While nosql creates uniform access performance by keeping the interface simple and scaling out, it also does not allow computations to be automatically pushed down to an individual node for parallel processing. That’s where hadoop steps in. By teasing apart the computing part from the data access layer, you have to now choose where computing occurs. In the case of hadoop, that processing can occur on a node where the data lives (there is a Cassandra+Hadoop integration layer) or you can process the data controlling using uniform access performance to avoid overloading the compute server. This also means there are really not any stored procedures in nosql databases.
      • Eventually Consistent: In our specific case, eventually consistent is fine. Since we are loading the data and deduping and cleansing it initially, that’s not a big deal. But…
        • Let’s also think through the case of Provider MDM. In an enterprise MDM system, all transactional systems should reference, in real time, the MDM system to obtain authoritative data when it needs it. The MDM data should be consistent. However, even in Payers today, the MDM data is not immediately consistent. There is an acceptable lag between one transactional system authoring some master data and another system in being able to access it. Typically, the lag is a few hours or a day or a few days. In fact, if we look at nosql databases like Cassandra, its quite possible to improve the time to consistency at a significant lower cost structure. For example, a social media site can tune its consistency which means it can tune how fast you can see your new “friends” post. You will also want to tune the consistency you want for MDM and scale it up or down. You can do this in nosql technologies without incurring additional development time or complexity all using the same database. That’s huge and compelling. Because enterprise MDM makes the MDM system an operational imperative, you have almost immediately solved some very vexing architecture problems at an incredible inexpensive cost point.
        • Cool architecture: The previous bullets have already pointed out the need for scale and robustness so I will not repeat that here. But an additional thought may be worth pointing out. If you think about the data access patterns, where the architecture is concentrated to have an MDM hub that selves up authoritative data to many, many consuming applications (many reads, few writes) and this all happening in relatively real-time, then scaling and fault-tolerance are actually key. In the MDM vision landscape, cool architecture is actually really important and your MDM hub does look more and more like a website serving up data. You need a transactional OD.  Its also important to realize that you can only take advantage of the cool architecture if the other parts of your architecture are also simplified. Your mileage may vary if all you are doing is plugging in new technology into the same old landscape without any changes anywhere.
          • Because a cool architecture can be delaminated, we have to plan for how computations (queries) will be executed. You cannot automatically have the computations pushed down to a node without using hadoop or something similar. Otherwise all computation and IO gets throttled on the node you issue the query from. That’s one are of review and choice you have to think through and one place where hadoop, hive and pig try to help you think through. Other database engines that distribute the data and computations might make this much easier but you may have to make other architecture choices to use those tools. Think deeply and carefully about cool architecture.

So based on some deeper thinking, it appears that the bigdata and nosql world can offer something of value even for a Provider MDM problem which seems like an ill-fit to begin with.

Summary

It appears that if the problem you are trying to solve is important enough to use these other technologies, there is some benefit to using them in the right mix and in the right proportions to your existing architecture. They are viable and based on our experience at Ajilitee, can be made production ready. In some cases, they can dramatically reduce operating complexity despite their seemingly lack of maturity around tooling. In many Payers, reducing operating complexity is a huge win.

In the next blog we will demonstrate learning by doing using bigdata technologies on larger Provider datasets and common healthcare processing analytical patterns. I’ll also return to the FWA theme.

As a treat, John Bair our CTO is speaking at TDWI’s Cool BI Forum in Chicago on May 8. He’ll be talking about these technologies and how they can help you. His talk is based on direct experiences from building products and solutions for our clients.

When the Great Communicator made that proclamation, a wall fell. Within months, the world became a different place.

In many organizations, a more resilient wall stands strong. Leveling the divide between business and IT is no mean feat. In fact, technology created and defined that divide. In somewhat of a twist of fate, technology has surprisingly become the enabler, the communicator. Technology available now can help drive collaboration in the business intelligence arena in ways we couldn’t imagine five years ago.

Data profiling is one area where tools get business and IT teams talking. My last blog discussed conducting joint data quality review sessions using data profiling tools as an accelerator. That approach enables a rich conversation between business and IT – a groupthink that creates a high degree of collaboration and trust between the parties that produce and consume data.

Similar collaboration tools are available for monitoring ongoing data quality, maintaining master data, building business glossaries, validating business rules, and other tasks. The tools leverage maturing technology and clever design to enable and enrich the tasks of stewards. These stewardship activities have been difficult to define and harder to implement – until now.

The clarion call here is not about simply deploying new technology – we’ve all been down that road before. Neither do our goals include making every business user a tool jockey. We’ve missed the point if we haven’t grasped the incredible attention to detail required to smooth over the business-IT divide. Tools used with patience and care gracefully handle these kinds of details. Our goal is to obscure the line between technology and business, while eliciting the responsibilities of stewardship.

Using such tools sets the organization into a virtuous cycle of continuous data quality improvement. The business users work with intuitive interfaces that mask the underlying complexities of data, master data and metadata. The tools simplify presentation of information, evaluation of options, and acceptance of inputs. IT gets the feedback needed from the business to improve data knowledge and quality. Players on both sides of the wall gain a deeper appreciation for the need to communicate effectively, while managing data as a corporate asset.

We can turn our organizations into entirely different places by leveraging these capabilities to systematically chip away the divide between business and IT.

This wall, too, is destined to fall.

Tuesday, May 8, 8:00-9:00 a.m.

Speaker: John Bair, CTO, Ajilitee

Advances in technology are leading us to rethink our whole approach to BI – what data we can manage, how we manage it, and ways to use the resulting information. In this session we will explore three of the most important trends in BI: big data, advanced analytics, and cloud computing technologies – and the implications to your BI and information management programs.

You don’t have to be an e-commerce company with petabytes of data to take advantage of these technologies. If your business or IT organization, like most, perceives that your data keeps growing, that your legacy BI platforms struggle to keep up, or that the lack of information is a business disadvantage, then big data, analytics, and cloud should be on your radar, regardless of the maturity of your BI programs. Although widely hyped, advanced analytics, big data and cloud computing are nonetheless emerging as tools to enable new applications, with lower barriers to use. As a result, these technologies are driving a renewed visibility of BI in the minds of business leaders. This session will share lessons learned by early adopters. It will discuss applications, architecture alternatives, key challenges encountered, and the resulting implications for BI strategies and programs aimed at transforming businesses of all sizes.

You Will Learn

• Why big data, advanced analytics and cloud computing technologies matter
• Architectures for building new business analytic services
• Strategies for moving from passive to active analytics
• Approaches for helping your business stakeholders lead with BI

Visit the conference website>>

Speaker: Tina McCoppin, Partner, Ajilitee

June 25-28, San Diego, California

Data Stewards have it tough. Pulled in multiple directions, they often juggle too much and fall prey to scattered responsibilities without generating lasting, impactful results.  Similar to being on a yo-yo diet, Data Stewards often find themselves either going strong or losing steam.

To strike the right balance of effort for sustained results, we recommend a Data Steward Health Plan that blends both cardio and weight training for data stewards to go the distance in their role.  In other words, Data Stewards should blend a focus on standards and conformity, metadata, enterprise glossaries and data dictionaries (“cardio”) with strength training to develop the skeletal muscles of your organization in such areas as enterprise data integration (EDI), data quality (DQ), and master data management (MDM).

This pump-it-up session will pinpoint activities that really count for Data Stewards. The Data Steward Health Plan is designed to trim the time wasters (or “fat”) and build cardiovascular endurance and muscle strength for optimal efficiency and results. We also will cover the latest specialized equipment (tools and frameworks) needed to target specific muscle groups and types of (data) movement to support your Power Data Steward training.

This session will detail:

• Primary/secondary responsibilities of a Data Steward
• How and where to “trim the fat”
• The Data Steward Health Plan: a week-by-week, month-by-month program to shed the weight and power through your job
• Practical tips for time management
• Must have tools for transparency and visibility
• Real-world examples

Visit the conference website>>

The right combination of proven quantitative and qualitative metrics will help improve your information, and also will drive continued executive support for your program.  Please enjoy our complimentary guide.

DOWNLOAD:  The Ultimate Guide to Data Governance Metrics for Health Payers:  30+ Ways to Discover and Score Success in 2012

Ajilitee is committed to ensuring that your privacy is protected.  Please view our full privacy policy here.

Nashville, TN

Visit the conference website>>

Vendors have come to market with toolsets that make it virtually inexcusable to run massive manual SQL checks to profile data. These data quality analysis accelerators provide an effortless and consistent set of data heuristics at the click of a mouse. The tools offer an ad hoc capability to see the data that is both broad and deep.  Here are just a few of the items that can be validated: unique values, domain and range of values, default values, data types, field formats, outliers, codeset validity, presence of nulls, blank data, and invalid characters.

The data profiling tools are the accelerator, but the real value comes from the meeting of minds at data profiling review sessions.

Data profiling review sessions need a quorum of business and IT participants. The analyst(s) who wrote the source target mappings and data requirements needs to attend. Invite business SMEs that know and use the data regularly. A QA representative should attend to clarify issues, log the issues into a tracker or enterprise quality tool, and track issue resolution over the coming days.

Sessions are generally guided by the analyst. The source target mappings and business requirements documents should be close at hand during the session for reference. The most useful review sessions have live connections to the data profiling tool and to the data sources. Questions that pop up during the review sessions can be addressed in more detail by drilling into the data profiler, and/or by querying the source data.

Send out the link to the data quality profiling results at least a day before the review session so that everyone has a chance to do reasonability checks. Distribute a checklist of generic data quality pointers. The checklist will direct attention to key fields like primary keys and fields required by the business reports.  This will also help ensure a consistent approach to the effort.

Witnessing the groupthink in these sessions is fascinating. Each participant comes to the meeting with a unique point of view on the project inputs and outputs. The data profiling results are parsed to answer questions, generating new questions. A quick exploration back into the source system provides some immediate answers. The data mappings and business requirements are validated.  New business rules are proposed on the spot to solve observed issues. The roundtable discussion can be rich and fruitful. This is one forum where the whole is certainly greater than the sum of the parts.

Automated data profiling has become so effortless that we need to consider checking data quality at many points in the development lifecycle – during source analysis, when source data is landed, and after business rules are applied. The data profiling exercise need not be limited to single files or tables. Many profiling tools have inter-table profiling capabilities that can help validate referential integrity and find orphan keys.

Analysts should be analyzing data profiling output, not writing and running endless queries. The delivery team reaches a new level of collaboration when the tools and processes to enable data profiling are part of the team mindset.

Founded in 2008, LaunchPoint has served 30 customers through its two divisions: Ajilitee, a consulting and services firm that specializes in agile analytics, business intelligence, cloud enablement, and information management; and Discovery Health Partners, a provider of intelligent healthcare cost containment solutions that include subrogation, coordination of benefits and dependent eligibility verification.  Customers include leading healthcare payers, provider networks, academic medical centers, and self-insured corporations.

Read full press release here>>

BI power users build complex reports, drive statistical models, satisfy ever changing regulatory reporting, and drive sophisticated analysis for your management team. These folks do the heavy data lifting at your organization. They are the data gurus, good ones are hard to find, and they don’t work cheaply. Without a sizeable contingent of these brainiacs, you’re stuck in a world of basic operational reporting. With them, you can rule the data that drives your business and sound decision making. You don’t have to wish for power users. Create them. This blog series will explain how.

There are many types of thinkers working in your organization. The people who make up your analytical community are no exception. Brain science splits learners into two camps – the visual and the logical learners, corresponding to the right and left hemispheres of the brain. The way to train your analysts is to cater to how each of their brains operates by bringing a little science into your tool selection, user enablement and training.

What is the right approach? You could survey each analyst to discover their learning style, then craft an optimal learning path with customized toolsets and techniques. That approach is likely to be slow, painful and expensive. Instead, tap into their instincts by leveraging their natural learning style. Latch onto their brains.

What does a brain-centric approach to creating BI power users look like?

As a visual thinker, SQL training threw me into a proverbial rabbit hole. I was set back months in my development as an analyst. Several years later when I started training end users on tool use, I thought everyone wanted to start out building querying skills using visual tools. I was fooled – twice – maybe you have been too. My predilection for visual tools is just as disabling a bias as a focus on pure SQL. We need to work with both camps. Truth be told most of your users do prefer starting with visual tools. On the other hand, some of your users are already comfortable with visual tools and need to work with code to enable more productivity. Forcing a right brainer into code too quickly, or a left brainer into image based analysis, is a mistake that is counterproductive to human growth.

The right mix of tools and support plays to both sides of the brain to enable your users to grow at an optimal pace. Visual enhances code, code explains and reinforces visual. Access to both types of tools, as well as the hybrid visual tools that allow code display – will please all your fledgling brainiacs. The better your development program accommodates their brains, the quicker you’ll see results. The ‘secret sauce’ is recognizing the delicate dance between images and words that deepens understanding. Power users’ brains work with a rich set of visual and coded tools.

I propose that the reason it is so difficult to grow BI power users is that we continue to ignore differences in how people think. We deploy tools using data requirements, not people requirements, and certainly not thinking requirements. Don’t ignore the differences. Grow your analysts incrementally by bringing their visual and logical parts into lockstep.

In my next blog, I’ll trudge deeper into this topic to explore how the right triage of tools and capabilities can help build that elusive BI power user community.

Move the code into the test environment after it has been thoroughly system tested.

I’ve never understood why projects rush to get code into the test environment. Think of the constraints on our working environments. Development is owned and controlled by the developers. Test, on the other hand, is locked down, typically by the QA team (and by all rights should be).  Working through functional canary tests in development is quicker, easier, and cheaper.

Execute the vast majority of system tests on the development integration platform.

A common practice is to migrate work from developer “sandboxes” to a development “integration” area, where code is assembled into a BI solution. The integration area is the ideal test platform. That move is an at least an order of magnitude easier than moving the code out of your freedom loving development environment to the lockdown on test.

Make system testing a formality.

Development of system test materials, scripts, and the test harness needs to coincide with the arrival of unit code into the development integration environment. Require the development team to demonstrate successful system testing within the development integration area BEFORE moving the code into the test area. The goal is to make system testing a formality. (Mindful readers will note that I stated just the opposite in part #5. The rule only applies if you complete the majority of system testing on development).

Regression test by design.

Too many projects focus on passing failed tests while ignoring previously passed tests.  The fallacy is that testing is complete when code fixes pass a retest. The reality is that code fixes can impact code that ran successfully in the past, causing test failure. Every set of code fixes mandates a test rerun.  Fortunately, our automated and fast-running functional canary test datasets means you can afford lots of test cycles (see part #6).

Every incremental code change requires the addition of a small set of tests to the functional canary test dataset. The entire test dataset runs together to verify that the incremental change and all prior code perform as expected. Each functional canary test set run validates the entire code base. This ‘black box’ treatment of the code (see part #2) ensures that every logic path – old and new – is retested each time the code is changed.

This is regression testing by design.

System testing expands to encompass new rules as the code base matures. Our functional canary data sets and scripts expand in scope and precision as the code base matures. The code base and system testing mature in lockstep, all outside the confines of the test platform.

Rerun your test scripts on the system test environment to validate code promotion.

Typical project plans deliver system test scripts and data sets just in time for system testing. Initial migration from development to test uncovers defects related to migration. Errors are found in the setup of system test data sets and test scripts – even if the code is working perfectly. This happens when neither the migration process, the test setup, or the code base are isolated. Discovering the source of issues is going to take time – and yet more migrations. This is an expensive way to validate testing and migration.

Let’s say the code passes all system tests prior to migration to the system test environment. You will know (in development) how well the code is working. The test scripts will be debugged before they reach the test environment. After promotion, repoint the test harness to the new environment and the right datasets and you’re ready to rerun system testing on the test box. The system tests will now focus on the validity of migration and setup of the test environment.  Any differences can be attributed to those issues, not bugs in the code or test scripts. This can reduce from weeks to days the time it takes to validate that the migration process works correctly. This is a cheap and expeditious way to validate migration.

Success is in the timing and coordination.

System testing done this way requires some TLC and more attention from management.  Development of the test scripts and the test harness need to be timed with the development and integration of the solution components. More coordination is required between developers writing code, creating test cases and test data.

Rethinking system testing

The testing process described in the seven parts of this blog may be seem counter-intuitive, given that what is proposed is the completion of most system testing before code touches the test box. You’ll need to rethink how your team conducts testing, to consider the content and timing of test scripts and data sets, to offer guidance to the development and testing teams, and to find a skilled developer to tie the tests and validations together with appropriate automation (see part #6). None of these are beyond the skills of an experienced development team working with a good project manager.

This approach to system testing can help hit project objectives and deliver the project sooner, to the kudos of management and the end user community.

Good luck with your system testing efforts.

Please share your thoughts about this blog, or relate your own experiences.

- Jim Van de Water contributed to this blog.