David Greenwalt
Steven Riney

Healthcare organizations need to achieve value from every investment-including IT. But how is that value measured?


At a Glance

Implementing a successful IT benefits measurement program consists of four steps:

  • Identifying benefits and metrics prior to project funding or approval
  • Determining baseline measurements
  • Performing a benefits opportunity analysis
  • Performing a post-implementation benefits audit

"It is not the goal but the way there that matters-and the harder the way, the more worthwhile the journey."
-Sir Wilfred Thesiger, British explorer (1910-2003)

Sir Wilfred's words are clearly not true in the business of health care. Goals are definitely what matters. Faced with inadequate margins, insufficient payment, and growing demand for services, healthcare organizations need to achieve value from every investment. That is just as true for IT as it is for every other investment in the organization's portfolio.

Yet IT is often seen as a black hole that consumes resources with few tangible results. Too many projects either fail or finish by delivering less than promised for a higher cost than expected. In addition, many executives do not have positive experiences with IT investments or have not been personally involved in the IT investment process. Most important, too few IT investments are evaluated within a benefits realization framework.

In a recent CIO survey conducted by IBM, the most frequently cited reasons that IT-related investments fail to deliver value include:

  • Poor alignment of processes and technology. Unfortunately, IT solutions often address processes that are only tangentially related to a problem rather than to its root cause.
  • Inadequate process change. Many organizations invest in IT but do not change their underlying business or clinical processes. Too often healthcare organizations implement new systems in the image of the old system and simply maintain existing, flawed processes.
  • A lack of executive ownership and accountability. Even if an organization properly aligns processes with technology and changes its processes accordingly, real benefits are rarely achieved without executive commitment to the project, clear accountabilities for achieving the benefits, and a framework for measuring the expected benefits. Because many well-publicized IT failures relate more to cultural issues than technology challenges, organizations need to manage technology, culture, and change processes.

Implementing a successful benefits measurement program consists of four steps:

  • Identifying benefits and metrics prior to project funding or approval
  • Determining baseline measurements
  • Performing a benefits opportunity analysis
  • Performing a post-implementation benefits audit

Identifying Benefits and Metrics

This first step requires an organization to identify the key benefits it expects to achieve from implementing the new solution and which metrics will best determine whether the project has achieved its goals.

The benefits selected should be those that most closely represent the organization's strategic direction. Healthcare organizations typically map strategic goals to more concrete objectives, which in turn are supported by specific initiatives. These initiatives are where specific benefits are likely to be achieved.

For example, an organization's strategic goal may be to enhance the patient experience, while a supporting objective might be to streamline the patient scheduling and access processes. A patient portal that enables patients to request services and check in remotely is an example of a specific initiative that supports streamlining patient flow. Although there are many benefits to such an initiative, patient satisfaction may best measure the effectiveness of achieving the original strategic goal of enhancing the patient experience.

This example demonstrates the flow of strategic goals to objectives and ultimately to initiatives. It also demonstrates the importance of measuring "soft" benefits. Although there may be quantifiable "hard" financial benefits through reductions in staff by enabling self-service, patient satisfaction more clearly measures the organization's effectiveness in enhancing the patient's experience. With consumerism on the rise, it may be this initiative that produces future volume and revenue gains, although it would be very difficult to specifically attribute cause and effect to those efforts.

Even though it is important to measure non-financial benefits, it is equally important that healthcare organizations not try to measure too many benefits. For every initiative, process, and workflow, many benefits could be measured, but only a few are truly important to the organization's strategic objectives. Organizations that try to measure too many benefits are vulnerable to metrics overload. Many benefits that various initiatives may generate are largely interrelated; trying to measure and monitor all of them would make it virtually impossible to analyze variances in expected performance. Ideally, even the most complex projects should have no more than 10 to 12 identified benefits to measure their effectiveness.

While a benefit is an improvement in operations such as patient care service delivery, a reduction of medical errors/risks, or a reduction in supply expense, a metric is a defined calculation designed to measure the results of a process. In the patient satisfaction example, a patient satisfaction survey score is the most likely metric to effectively measure the benefit. 

Determining Baseline Measurements

Once the benefits and metrics have been identified, a baseline for each metric should be calculated. As part of this process, it is necessary to review the data and data sources that are needed to establish each baseline metric. This process serves both to develop baselines and to act as a feedback loop for the prior step.

These benefits and metrics might be ideal for measuring the effectiveness of an IT solution. However, the organization's ability to calculate that metric may not be apparent until it attempts to determine the baseline value. Identifying benefits and metrics is often a strategic activity; however, the actual work of calculating the metric occurs in the trenches. It is possible that the team that identified the initial benefits and metrics may not be aware of data collection challenges at a tactical level. The process of evaluating data sources and availability provides an opportunity to re-evaluate the previously determined metrics.

Data resources will be available for those metrics that survive this step. It is possible that there is a forward-looking metric for which a baseline cannot be calculated using current processes. Most organizations cannot objectively measure adverse drug events using today's technology. Without the advanced clinical systems that these organizations are installing to mitigate this problem, real baseline data collection is problematic. Facilities without an automated system collect this information on a self-reported basis. Comparing self-reported data with rules-based calculations post-implementation will likely show inappropriately high rates of errors with the use of the new automated system-exactly the opposite of what was intended. In reality, only the capture of the error has improved and now the reporting of the metric is more accurate. In these cases, the baseline measure should be delayed until the initial information can be gathered using the new solution.

When good baseline measures are available, organizations should develop methodology and definitions in addition to baseline values. Although many calculations are standard throughout the industry, there are local, state, and regional differences in some statistical calculations, often resulting in a "you say tomayto and I say tomahto" scenario. Clarifying the definitions and calculations of each metric while determining the baseline value will help mitigate discussions post-implementation during the benefits realization phase. 

Performing a Benefits Opportunity Analysis

Once the organization has determined what it wants to improve, the natural question is, how much can we improve? This is the most complex step in the process and requires a thorough understanding of the organization and its culture.

In a perfect world, best practice data would be available for every process within an organization, and every organization would be able to implement best practices with enabling technology and achieve the ultimate results. However, in reality, there are few standards for best practice. There are local regulatory, demographic, and cultural issues that make standardization difficult. Even in these situations, there may be standards or other published data that can determine the "real" opportunity from a potential IT investment.

To determine the opportunity for achieving any individual benefit, an organization should: 

  • Identify any national standards or benchmarks
  • Review any published studies by similar organizations
  • Evaluate the methods used to obtain these results
  • Balance the above with the organization's own culture and capability

In some areas, national standards and benchmarks are universally applicable and should be consistently used. Many clinical standards are universal, but many operational standards are situational. As an example, the patient experience is much different in a rural community hospital from that in an urban specialty care center. Therefore, it is important to review standards and literature in the desired areas of improvement and filter the information through the reality of one's own organization and culture. Although a national standard or a published study may indicate that a particular metric value is attainable, that value may be inapplicable for another organization for any number of valid reasons.

When balancing the empirical data with the organization's own capability, it is important to identify the individual who will be accountable for realizing the benefit. One of the most important factors in determining the scale of potential improvement is a commitment from the individual who will be responsible for achieving that improvement. It is important to review standards and literature, but each project ultimately hinges on individual accountability even when it is through a partnership arrangement between the operating executive who will focus on process and cultural changes and the CIO who will target more of the technical challenges. A partnership of both parties allows each individual to gain a broader understanding of the desired outcomes and appropriate methods to drive success. 

Performing a Post-Implementation Benefits Audit

Many healthcare organizations start down the path of a benefits measurement program but don't complete the task. They may identify benefits and metrics just to get board approval for a project. Or they may develop benefit opportunities that support detailed business plans, but those plans often gather dust on a shelf. Perhaps the holy grail of assessing IT investments is the post-implementation benefits realization audit. The real value in building a plan is measuring against that plan and learning from the differences.

The previous steps involve developing a plan according to what is expected from an IT-related investment; the post-implementation benefits audit provides valuable feedback as to variance from that plan. Variances provide information; they are not "grades." The information can be about the implementation or the process used to develop the plan. In any case, the learning and the ability to apply that learning to the next investment drive a continuous quality improvement approach to IT-related investments. 

The Need for IT Governance

IT governance is the systematic process of deciding who makes what IT decisions. It also includes the underlying processes that support the effective prioritization and transparency of decisions concerning IT-related business investments. IT governance is the systematic analysis, prioritization, and realization of benefits from all IT-related project investments through consistent governance processes.

Many organizations have tried IT steering committees to rationalize their IT-related investment priorities but have staffed them with technically adept physicians, key managers, and a few executives with no power to fund or hold project sponsors accountable. Committees of this nature often fail, sometimes dying a slow and agonizing death. Governance is not effective if many of the "real" decisions are made outside the IT governance process through political maneuvering or "screaming the loudest." A few other indicators of failing IT governance are persistent, nagging questions as to IT value; managers who cannot accurately describe the project approval process; and decisions that are made through any process except the formal approval process.

Key indicators for IT governance success include executive ownership, effective partnerships between the CIO and other executive project sponsors, and clearly assigned accountability for cultural and workflow issues at the executive table for each project. An effective IT governance process changes the perspective from an allocation of capital to an investment of capital with an expected ROI that needs to be managed, understood, and supported by key executives to consistently deliver value from the organization's IT-related investments. 

Managing the IT Investment Portfolio

Healthcare organizations need discipline, consistency, and executive ownership if they are to resolve the questions of IT value, both at the project level when IT-related investments are funded and from a global perspective of managing the IT-related investment portfolio. Managing that portfolio is not much different from managing the organization's other investments when the process begins by identifying metrics and finishes with a consistent benefits realization process for each investment.


David Greenwalt is a managing consultant, health analytics practice, IBM Healthlink Solutions, based in Denver (dgreenwalt@us.ibm.com).

Steven Riney is a managing consultant, strategic services practice, IBM Healthlink Solutions, based in Modesto, Calif.


Sample Metrics for Measuring Effectiveness

Following are some metrics examples that might be used to measure the effectiveness of a new pharmacy or scheduling system.

Pharmacy

  • Adverse drug events rate
  • Average number of minutes to complete medication orders
  • Formulary compliance rate
  • Average pharmacy charges per patient visit
  • Average medication orders per patient visit

Scheduling

  • Average patient wait times
  • Patient no-show rates
  • Scheduling cost per scheduled procedure
  • Patient visits per physician per day (throughput)

Publication Date: Thursday, February 01, 2007

Login Required

If you are an existing member, please log in below. Username and password are required.

Username:

Password:

Forgot User Name?
Forgot Password?







Close

If you are not an HFMA member and would like to access portions of our content for 30 days, please fill out the following.

First Name:

Last Name:

Email:

   Become an HFMA member instead