Douglas I. Thompson
Neil S. Fleming
Financial data in clinical studies that document the value of electronic medical records (EMRs) must be adjusted for use in ROI calculations to provide credible estimates of EMR benefits.
At a Glance
There are five common pitfalls in using clinical studies to calculate the ROI of electronic medical records (EMRs):
- Imputing value to minutes of time saved when staffing is not reduced
- Imputing or estimating cost savings that can't be measured
- Ignoring the revenue impact of reduced resource utilization
- Ignoring baseline performance in extrapolating benefits
- Using fixed costs in financial savings analyses
American hospitals and physician practices are spending more on clinical information systems than ever before-and the fastest growing segment is "advanced" electronic medical record (EMR) systems. Computerized provider order entry (CPOE), CPOE-driven decision support, automated clinical documentation, electronic patient records, clinical data repositories, integrated ancillary systems, and other more limited tools and capabilities are now sold as integrated packages by a growing number of vendors.
The industry has apparently reached the "tipping point" at which the expected benefits of these systems have exceeded their anticipated costs and risks. In some circles, the decision to purchase an EMR is considered to be a "no brainer." The belief is that an EMR will deliver substantial clinical and financial benefits. Purchasing an EMR system also is widely considered "the right thing to do," based on the premises that it's a cost of doing business in today's healthcare environment, that physicians and nurses expect such tools, that the use of EMRs will someday be mandated, and that it is a provider's duty to patients to ensure quality and safety.
However, there is a large gap between popular concepts of EMR value and the evidence available to support estimates of the amount of that value. In other words, there is a difference between what these systems have reportedly been designed to do and what they have actually been able to do thus far. This discrepancy puts pressure on hospital CFOs and financial analysts who must justify the use of these systems from a financial perspective.
Studies Don't Tell the Whole Story
There are three major challenges in using published information to justify clinical information systems. These challenges are the lack of comprehensive studies, the lack of a common basis for comparison, and the inconsistency of financial information between studies.
Lack of Comprehensive Studies
There are very few comprehensive studies of hospital EMR value. Hospital staff therefore must identify, obtain, compare, combine, and extrapolate the results of numerous focused studies to assemble a complete picture of EMR value.
For example, a PubMed search covering the past five years, using the search term "hospital EMR benefits," yielded 83 articles. The diverse types of information contained in these studies are shown in Exhibit 1.
Only nine of the 83 studies included quantitative data about hospital EMR benefits. What's worse, only two of the nine studies with quantitative data discussed hospitalwide EMR benefits. The other seven studies explored only certain subsystems or focused areas of benefit.
Other researchers have found the same results. For example, a 2005 RAND review of 256 published studies, Economic Value of Electronic Health Record Systems and Health Information Technology Applications, attempted to quantify an EMR's economic value. Although 82 of these studies considered the hospital inpatient setting, not one rigorous study was found that quantified the economic benefits of a full-functioned, vendor-supplied system.
Lack of Comparability
Studies of similar system functionality and/or benefit types lack a common basis for comparison. Most of these studies use different data sources, research methods, and metrics. Hospital organizational structure, processes, culture, baseline performance, system functionality, and implementation approaches are also different. All of this has an impact on reported results and the ability to compare and extrapolate these results.
Consider the partial list of references to EMR-related drug use and cost reductions (see Exhibit 2) taken from a 2007 study (Thompson, D.I., et al., "EMRs in the Fourth Stage: The Future of Electronic Medical Records Based on the Experience at Intermountain Health Care," The Journal of Healthcare Information Management, Summer 2007).
Although each of these studies refers to drug cost reductions, the individual drugs or categories of drugs mentioned are different, the level of impact the systems had is different, and the metrics used to quantify the systems' impact vary from utilization to costs to charges. Given this amount of variation, it's important to carefully read and understand the full text of each study to know how to compare studies and which one(s) to weight most heavily in estimating your own benefits.
Lack of Standard Approach
The financial calculations of study results contained in many published studies must be substantially adjusted to avoid arriving at wrong conclusions about the value of an EMR. There are five common pitfalls into which the financial portions study results fall, and the data can invalidate the resulting benefit models if they are not reversed or otherwise adjusted.
Pitfall No. 1: Using fixed costs in financial savings analyses.It is common practice in many published benefit studies to use full costs instead of variable costs for savings calculations. Under most realistic operating conditions, only variable costs would be saved, so this practice greatly overstates expected benefits.
Information from two well-known and often-cited studies of adverse drug event (ADE) costs is shown in Exhibit 3.
Both of these studies used total costs rather than variable costs from their hospital cost accounting systems or cost-to-charge ratio calculations. Adjusting to variable costs is essential to obtaining a more realistic estimate of potential ADE-related savings. Nursing costs do not vary when a single ADE is prevented on a given nursing unit, since the absence of a single patient from a nursing unit-even an ICU, in most cases-does not change the number of nursing staff required. According to a 2007 First Consulting Group nursing survey (unpublished), nursing costs are approximately half of the total costs of an ADE, so using the Classen figure leaves $2,148 in remaining 2007 costs, including laboratory, drug, and other testing and procedure costs. Variable costs are approximately half of total costs at a typical U.S. hospital (Lichtig, L., et al., "Measuring Clinical Effectiveness Using Common Hospital Data," P&T, May 2004), so the 2007 variable cost, or cash benefit, of preventing an ADE is estimated at $1,074. Using the total costs from the Classen or Bates studies would have resulted in unrealistically high estimates of cost savings from ADE prevention.
Pitfall No. 2: Imputing financial value to staff time savings. Another common practice in published benefit analyses, and those supplied by vendors and consultants, is to calculate the amount of staff time saved by using an information system, and to impute a financial value to that time. In many real-life examples, staff time savings do not result in staffing reductions. Instead, other tasks are found for the staff to perform. However, the benefit calculation is often done as if salary costs for the time saved were eliminated. Unless staffing is actually reduced, there are no salary cost savings. An acceptable alternative would be to explicitly value the additional revenue or other financial benefit produced by new staff activities.
Example: A recent study found that time saved through the use of nursing documentation systems could be used to "take care of additional patients, keeping quality constant," and that this would translate into "a reduction of the demand for nurses" (Girosi, F., et al., Extrapolating Evidence of Health Information Technology Savings and Costs, RAND, 2005). Based upon their evaluation of three studies, the authors estimated that nursing documentation systems could reduce the U.S. national demand for nurses by 11.4 percent, and save an average of $7.1 billion annually (equivalent to $2.2 million annually for a typical 300-bed hospital).
This study confuses documentation time savings with staffing and cost savings.
A recent meta-analysis of 11 studies comparing nursing documentation time on paper with electronic systems found that nurses who used clinical documentation systems saved 24 percent of their documentation time, on average (Poissant, L., et al., "The Impact of Electronic Health Records on Time Efficiency of Physicians and Nurses: A Systematic Review," Journal of the American Medical Informatics Association, May 19, 2005). A review of most of these 11 articles individually disclosed that a 24 percent time savings was equivalent to 28 minutes to 36 minutes per nurse, per eight-hour shift. The total time savings on a typical nursing unit is not enough to send even one of the nurses home, so nursing documentation time savings are unlikely to reduce nurse staffing.
Pitfall No. 3: Imputing cost savings that can't be measured.When an information system prevents adverse events, there may be a financial savings, as the costs of repairing the injuries caused by the event are avoided. The incidence of adverse events can be measured before and after system implementation, but the actual cost of the events that are prevented is unknown, because it's impossible to measure an event that didn't happen. This cost can be imputed by carefully comparing cases that included adverse events with similar cases that did not include such events, but the result is still an estimate, not a measurement, and this technique is beyond the practical capabilities and/or budget of many hospitals. This means that, in practical terms, it's impossible to budget for, assign responsibility for, and actually measure specific cost reductions from adverse event prevention.
Hospitals should carefully consider how they handle estimates of financial benefits in this area, since these estimates will not be able to be confirmed. The same advice also applies to other adverse events, such as infections, falls, and pressure wounds.
Example: Based on several studies estimating the cost of ADEs, Allina Hospitals and Clinics projected substantial financial savings related to ADE prevention associated with their CPOE implementation. These savings were included in operating budgets, and local hospital presidents were charged with achieving the budgeted cost reductions. But Allina executives soon realized that they had no way of measuring these savings, and the budgets had to be readjusted to remove the ADE-related savings (Pederson, K.A., and Henry, S., "Driving for Benefits: The Lessons Learned," conference call presentation sponsored by the Scottsdale Institute, May 31, 2007).
Pitfall No. 4: Ignoring the revenue impact of reduced resource utilization.Many of the cost savings associated with EMR implementation come from reduced clinical resource use. However, there is also a revenue impact from reducing utilization. Payments that are based on charges, lengths of stay, or costs will be reduced if, for example, fewer tests, drugs, or days of care are required. In some instances, these revenue reductions are greater than the expected cost savings. The revenue impact should always be estimated and measured along with the cost savings, in order to get a complete picture of the financial impact of an EMR.
Example: None of the six studies shown in the earlier exhibit attempted to estimate the impact on reimbursement from reducing drug utilization, costs, or charges. However, depending upon a hospital's payer mix and contract terms, these "savings" might actually wind up costing the hospital money. Exhibit 4 shows the impact of different payer mix assumptions on the net financial impact of drug cost reductions.
Pitfall No. 5: Ignoring differences in baseline performance. One of the reasons for the large differences in published EMR benefits is that the reference hospitals start from different levels of performance prior to system implementation. These differences in performance should be factored into EMR benefit estimates: If a hospital already performs excellently in a particular area, then the amount of benefit to be realized from an EMR is less than for another hospital with average or below average performance.
Example: A recent review of several studies showing the impact of an EMR on hospital lengths of stay (LOS) illustrates this point in four of the studies examined (Thompson, D.I., et al., "EMRs in the Fourth Stage: The Future of Electronic Medical Records Based on the Experience at Intermountain Health Care," The Journal of Healthcare Information Management, Summer 2007). The overall LOS reductions described in the four studies ranged from 2 percent to 30 percent, as shown in Exhibit 5, with greater reductions observed in subpopulations. A 5 percent to 10 percent reduction in overall LOS may be realistic for many hospitals implementing EMRs; however, it depends upon where the hospital is starting from. The exhibit shows pre- and post-EMR LOS for hospitals in the studies mentioned above.
As the exhibit shows, the greatest EMR-related LOS reduction came at the Korean hospital described by Hwang (3.2 days, a 28 percent reduction). On the other end of the spectrum, the two university hospitals described by Mekhjian showed 0.2-day and 0.07-day (5.1 percent and 1.9 percent) reductions. But the starting LOS at these two hospitals was less than four days, while the Korean hospital's was more than 11 days, so it's not surprising that this hospital realized a greater LOS reduction.
Overcoming the Pitfalls
In many cases, these pitfalls can be overcome to improve the usability of published studies in estimating local hospital benefits. Here are five principles to overcome the pitfalls.
Principle No. 1: Adjust savings estimates to variable costs. Find out whether the study used full costs or variable costs and adjust full costs to variable costs before applying them to your hospital. In some cases where variable cost percentages are not published, this will mean using a rule of thumb to make the adjustment. For example, it was assumed that variable costs were 50 percent of fixed costs for some of the calculations in a 2007 EHR value study, because actual variable cost data were not available (Thompson, D.I., et al., "EMRs in the Fourth Stage: The Future of Electronic Medical Records Based on the Experience at Intermountain Health Care," The Journal of Healthcare Information Management, Summer 2007).
Principle No. 2: Model potential staff reductions. Rather than relying on the article's calculation of cost savings from minutes of time saved, model the potential for actual staffing cost reductions using figures from your own hospital. For example, if the article says that 30 minutes of time per person was saved per shift, discuss this time savings with the head of that same department in your hospital, and determine with them whether 30 minutes per person would enable them to reduce staff or otherwise reduce costs. With the help of operational managers, it is not difficult to come up with realistic benefit scenarios. Be sure to discuss potential overtime cost savings, and other revenue-producing activities that could be completed with time saved.
Principle No. 3: Treat nonmeasurable cost savings differently. Each cost savings estimate should be matched with a metric and a method for measuring the actual benefit once it is realized. If no practical method of measuring the actual benefit can be found, then those benefits cannot be budgeted or assigned as an operational responsibility to an individual or department.
Principle No. 4: Calculate the revenue impact of reduced charges. Calculate the impact of lost utilization-based (percentage of charges or per-diem) or cost-based reimbursement.
Principle No. 5: Understand your baseline performance relative to published studies. Identify and account for differences in your baseline performance compared with that at the reference study site(s). Try to find hospitals that started with similar performance and a similar operating and technology environment. Adjust your expectations of improvement when the hospitals in published EMR benefit studies began with substantially better or worse performance than your hospital reports.
Take the Time to Do It Right
There is no substitute for the hard work required to construct meaningful, literature-based benefits models. This includes finding and obtaining the full text of the right studies, and carefully reading the full text of those studies in order to understand their relevance and applicability. It also includes careful adjustment of the results reported in these studies to avoid the common pitfalls described in this paper, and support meaningful estimates of EMR-driven performance improvements. This approach takes longer but results in better, more credible estimates and an improved ability to manage to the benefits, based on a realistic understanding of what the information system can and cannot do.
Douglas I. Thompson, FHIMSS, is a principal, CSC, Inc., McLean, Va. (firstname.lastname@example.org).
Neil S. Fleming, PhD, is vice president of healthcare research and director, Center for Health Care Research, Institute for Health Care Research & Improvement, Baylor Health Care System, Dallas (NeilFl@BaylorHealth.edu).
Publication Date: Tuesday, July 01, 2008