Healthcare Leaders Grapple with AI Investment Decisions at HIMSS25
At the HIMSS25 conference, healthcare leaders are intensely focused on integrating artificial intelligence (AI) into their organizations. With the market flooded with various AI solutions, health systems face a complex decision-making process: How to choose and implement AI tools strategically, given their significant costs.
Amidst the array of new AI tools, each touting impressive capabilities, leaders must meticulously evaluate the balance between cost and benefit to avoid investing in solutions with limited returns.
Separating the Hype from Reality
According to Nigam Shah, MBBS, PhD, the chief data scientist for Stanford Health Care, the cost-benefit analysis of AI tools extends beyond their core functions. For example, consider AI-powered predictive models.
“The model gives you a prediction or risk stratification, if you will, but the benefit comes from the response to action,” he explained during a HIMSS25 session in Las Vegas. “And so, in that setting, we have to study the model, we have to study the capacity to act, we have to study the cost and benefit of the action itself.”
For health system leaders, a critical question for cost-benefit analysis is determining the utility gained from a prediction. Depending on a health system’s ability to act on predictions, the utility could be measured as hours saved, money saved, or even lives saved. However, if the capacity for action is limited, the utility will diminish, reducing the tool’s ROI.
Consider a predictive AI model designed to identify sepsis risk in patients: It is ineffective if the health system doesn’t have enough clinical staff available to intervene quickly. Therefore, when evaluating new AI tools, health system leaders must carefully define and confirm benefits, as the benefit gleaned from a tool may differ from the anticipated one.
For example, Stanford assessed the impact of automated in-basket response technology on productivity, expecting time savings and reduced after-hours work. However, the results were different.
“Our doctors love it, their cognitive burden went down, and they’re happier,” Shah said. “100% true. [But] that’s not what we went in with. If you did an RCT in this way, the FDA would never approve your product because we changed the outcome after we finished the study.”
He also emphasized that applying the efficiency/productivity lens to generative AI may yield counterintuitive outcomes. Improving productivity might allow physicians to see more patients potentially exacerbating their workloads. Thus, health systems should focus on a tool’s “achievable benefit” when conducting cost-benefit analyses, Shah advised.
Exploring Different Approaches to Cost-Benefit Analysis
The benefits derived from AI tools vary depend on the specific use case. For example, physician well-being is a significant benefit for health systems implementing AI to alleviate administrative tasks. In an interview at HIMSS25, J. S. Smitherman, MD, Chief Medical Information Officer at Providence, noted that implementing AI across a large organization constitutes a substantial financial investment.
However, Providence’s cost-benefit analysis considers more than just productivity; they evaluate the ROI gleaned from keeping physicians in the workforce longer.
“Ambient [AI] tools make it easier for them to practice, and they’re willing to work for another year or two,” Smitherman said. “In aggregate, we have almost a quarter of the doctors in the United States over age 65. And with an aging population, that is a major crisis. We need to keep a lot of those doctors practicing if we want to have [healthcare access].”
It’s not only the large numbers of retiring physicians that are concerning Smitherman. He added that many younger physicians struggle to balance the demands of a full-time job with the time allotted for them. With nearly 50% of the physician workforce reporting burnout, ambient and other types of AI that decrease that cognitive burden may justify the cost.
The positive impact on physician well-being at Providence appears to be worth the price, with robust physician adoption of AI tools. Smitherman pointed out that adoption is a vital measure of whether technology implementation is effective.
“Doctors are sort of notoriously resistant,” he said. “So, it’s got to work. They’re going to vote with their feet. If it weren’t working, the doctors wouldn’t use it.”
Northwestern Medicine adopts a “fail fast” strategy inspired by Silicon Valley when conducting cost-benefit analyses of AI. Hannah Koczka, vice president of ventures and innovation at Northwestern Memorial Hospital, noted that the health system implements many technologies, testing them quickly on a small scale.
“It’s more like a proof of concept — something that we can maybe do in a matter of weeks or only do something in one department or it in one area before we then see if we can prove it out,” Koczka said. “And then we might do something at a much larger scale, like a six-month pilot or something like that.”
This quick approach to evaluating AI technology allows the health system to evaluate many technologies before committing to a tool that meets its needs and justifies the high cost.
Health system leaders also anticipate that costs may decrease as AI becomes more widespread. According to Smitherman, AI tools that address low-level administrative burdens, such as reviewing in-basket messages, are already less expensive than they used to be.
“As these [AI] models get better and faster, that’s improving,” he said. “But still, there’s going to be a lot of places where we do have to choose to make investments; but once again, because they’re different products, we just kind of have to evaluate the various ROIs. I think some of the ones where [there is a] clinical gain might be the toughest, right? Because nobody wants to put an exact number value on anybody’s life, but if an AI tool can potentially make something safer, the system certainly is willing to invest in that.”
Anuja Vaidya has covered the healthcare industry since 2012. She currently covers the virtual healthcare landscape, including telehealth, remote patient monitoring, and digital therapeutics.
