Five Sharper Questions to Separate Real Digital Health ROI from Wishful Thinking

As digital health becomes a staple of modern benefits strategies, employers are inundated with solutions promising better outcomes, higher engagement, and a strong return on investment (ROI). This is because digital health companies know that ROI is the buzzword expected in every benefits presentation. But while ROI promises span a broad spectrum of definitions and "soft" engagement anecdotes, they rarely offer money-back guarantees based on transparent and verifiable data. 

In a labor market where employee well-being is both a competitive advantage and a cost center, benefits leaders can’t afford to mistake marketing claims for measurable impact. Choosing the right partner means asking the right questions – about outcomes, populations, methodology and accountability.

As benefit leaders evaluate the credibility of driving a medical cost ROI, here are five essential questions every employer should ask before committing to a partnership:

1. Are they measuring engagement or actual cost savings?

It’s easy to be impressed by sign-ups, glowing user reviews, or app engagement. But these alone rarely result in actual cost savings. These engagement metrics must be combined with improved health outcomes and reduced claims to truly bend the cost curve. 

Avoided ER visits or reducing the need for expensive medications can have a significant near term impact. For long-term outcomes, programs that identify and address the root cause, support lifestyle change, and provide education geared toward self-management can reduce future health complications and associated utilization costs. 

In general, strong ROI should be tied to measurable, cost-reducing outcomes across care categories within a meaningful time horizon. In digestive health, for instance, conservative-yet-effective treatments that help patients achieve symptom control can prevent high-cost diagnostics. In musculoskeletal care, digital programs have helped lower surgery rates and reduce opioid use, often within the first year.

Smart Questions to Pressure-Test the Pitch:

• Which specific claims (CPT or NDC codes) does your solution aim to reduce in the near term and what are the implications over time? (It’s important to note the difference between restricting needed care vs reducing avoidable care - both of which can lower cost in the short term, but one can balloon later costs.)

• Do you have clinical trial studies showing actual claims reductions? Was the study done by the vendor, an independent organization, or the customer themselves?  

• How do you connect the dots between clinical improvements and financial impact?

• If ROI is only expected by Year 3, how should we assess value in the short term?

2. Does the math actually add up?

Vendors often make ROI claims that seem impressive—but when you look closely, the numbers do not withstand scrutiny. Many rely on vague definitions, hopeful assumptions, or take credit for outcomes they likely didn’t cause.

Some use questionable assumptions—like projecting results from a clinical study on specific populations to apply to all patients in the real world, focusing on selection bias to inflate data or state cost savings based on self-reported improvements that don’t actually show up in healthcare spending.

A credible ROI model should clearly explain how the program saves money—step by step—by preventing, reducing, or shifting specific costs. A good test: ask the vendor to walk through a typical member’s journey and show exactly where the savings happen. If they can’t, the ROI is probably more wishful thinking than achievable.

Smart Questions to Pressure-Test the Pitch:

• Can we review your ROI model with all assumptions visible?

• What’s the projected savings per member, and how was it calculated?

• What’s the assumed condition prevalence and appropriate engagement rate?

• Has this model been reviewed by a third-party actuarial firm?

3. Is the clinical evidence strong enough to justify the spend?

Even the best ROI model collapses if the underlying clinical impact isn’t real. Some vendors lean heavily on internal surveys or case studies. Others lean on vague metrics, such as productivity or reduced missed days, that are hard to measure directly. Clinical results should be grounded in peer-reviewed research or, at a minimum, conference-grade data with control groups.

Also, beware of cherry-picked data, where outcomes are driven by a small group of highly engaged users. Likewise, be cautious with before-and-after charts lacking a control—natural recovery and regression to the mean can be misleading.

Smart Questions to Pressure-Test the Pitch:

• Have your outcomes been peer-reviewed or presented publicly?

• What kind of control group was used? 

• Was there statistical comparative patient matching, or just baseline comparison?

• What was the n-size of the study population? 

• What’s the size and significance of the effect? What was the follow-up duration?  

• How do you ensure the data reflects your full population, not just your best users?

4. When—and for how long—will savings appear?

Finance teams operate on annual cycles, but health improvements often take longer. Misaligned expectations can lead to disappointment. That’s why it’s important to ask: when will savings show up, and how long will they last?

Three savings horizons to keep in mind:

• Near-term (0–12 months): Cost saving includes ER diversion/avoidance, imaging reduction, inpatient admit reduction, drug step-downs, some procedures or surgeries—quick to measure.

• Mid-term (12–24 months): Fewer surgeries or short-term disability days—requires trend analysis.

• Long-term (2+ years): Reduced disease progression, lower readmissions or complication rates—harder to track and often outside the average employee tenure.

Smart Questions to Pressure-Test the Pitch:

• Will you map each claimed savings to one of these timeframes?

• Can you align performance guarantees with near-term outcomes if short-term ROI matters most?

• Are you able to adjust expectations for long-term savings if your workforce turns over frequently?

5. What percentage of fees is truly at risk?

A 2:1 ROI “guarantee” tied to a 5% rebate isn’t risk-sharing—it’s marketing. True accountability means meaningful dollars on the line, not token concessions.

Some vendors risk just 5–10% of fees. Others are willing to stake 30–70% or more. The most aligned partners are confident enough to put 100% of fees at risk, tying payment to real, measurable outcomes that matter to you.

Smart Questions to Pressure-Test the Pitch:

• What portion of fees can be clawed back if goals aren’t met?

• Are KPIs tied to events that can be measured in claims?

• Who determines whether the vendor delivered, and when are refunds issued?

No More Smoke and Mirrors: How to Spot the Real Deal

A credible digital health partner doesn’t just promise results—they prove them. They measure success through claims data, not just app activity. Their ROI model is built on transparent assumptions and grounded in your population. Their outcomes are backed by real evidence. They tie savings to timelines your Chief Financial and Medical Officers understand. And they’re willing to put real dollars—ideally, all fees—at risk.

Anything less? It’s likely just marketing dressed up as impact. Choose partners who filter the noise, focus on what truly moves cost (ideally while improving care and experience), and are confident enough to be held accountable.

---

Nathan Paulsen is Vice President of Payer Partnerships at Oshi Health, a virtual gastrointestinal clinic delivering evidence-based care for the full spectrum of GI conditions, backed by a 100% fees-at-risk ROI guarantee.