Some major health insurance companies are using artificial intelligence to approve or deny claims.
It's raising concerns over human oversight and whether the technology can address preexisting disparities in the healthcare system.
"These days, in the age of AI, things have become much more automated," said Jude Odu, a healthcare technology expert with 25 years of experience. "It used to be that before an insurer denied a claim, a human being had to look at [the claim], look at all the clinical context. But then what has happened over the past few years is that they are now farming that out almost 100% to AI."
Odu has worked for one of the largest health insurance providers in the U.S. He also founded Health Cost IQ, a company that uses AI models to identify waste and inefficiencies in government- and employer-sponsored health plans, and published a book in May about AI-powered health plans.
ALSO READ: Health equity scorecard finds Black patients have the worst outcomes in Florida
Odu recently spoke with WUSF's Gabriella Paul about the dangers of leaving medical coverage decisions solely up to AI.
This interview transcription has been edited for clarity and length.
Well, Jude, you started your career working for United Healthcare. So, you've seen how medical claims decisions were made before AI entered this space. What was that like?
Yeah, so, back then I worked for the appeals and denials department, and although I wasn't part of the committee that did the denials, I was privy to the process where nurses would come in, the medical director would come in, they would look at the case and talk about it for maybe 30 seconds, and the medical director would simply put the gavel down and say, "Denied," and they will close that file and move to the other one.
So, that gave me a front-row seat to how our healthcare system works from the inside out.
You also have an international perspective on the U.S. healthcare system. You were born in Nigeria and grew up in Germany. What can you tell me about how our health insurance system stacks up globally?
Things are much different outside the U.S. in the rest of the developed world, if you want to look at it that way. People, for example, don't go bankrupt because of a hospital bill. That is so totally different from here in the U.S. where healthcare is big business.
Look at it from this perspective: Denials are actually good business because these are large shareholding companies. So the less you have to pay out in claims, the more profit you make. It's a very simple equation.
And what does it mean when you add AI into that equation, particularly when you think about preexisting health disparities and this notion that AI is known to scale errors rather than fix them?
The AI is only as good as the data it was trained on, and only as good as the people that wrote the computer program, right? So in cases of health disparities, what AI essentially does is, it takes the existing frameworks of discrimination, and it just magnifies them because that's what it's trained to do.
I have a few examples for you.
United Health Group bought a company called NaVi Health in 2020 for $2.5 billion. [It has] an AI algorithm. ... Now they're being sued because they're alleging that nine out of 10 predictions coming out of this system gets reversed when the patients appeal ... which means the AI is not doing a good job.
ALSO READ: The growing use of artificial intelligence in healthcare and implications for disparities
There was [another] AI system that is used to schedule patient appointments, and [it] produced 33% longer wait times for Black patients. That model [used] things like ZIP code, employment status, insurance type and past no-show rates — all of which correlate with race.
That's obviously problematic. So, is there an argument for AI in the health insurance space?
Yes. This all boils down to the right type of implementation of AI systems. That means that you have to consciously look out for the possibility that an AI system will amplify the preexisting disparities.
For example, you can set up an AI system that audits 100% of all claims and then [direct] it to detect instances where certain things [look] discriminatory in nature. For example, it can look out for when certain procedures are being denied at a much higher rate for specific ZIP codes or specific demographics than everybody else.
There are models that can be trained on an inclusive data set, which can identify members at high risk for, let's say, chronic conditions, and then those AI systems will do what they do best, which is orchestration. That means triggering events and actions to happen based on what it's finding. It can be used proactively to close gaps in care for populations.
That shift in perspective, in short, is saying the "North Star" for using AI in healthcare spaces should not be efficiency, but rather patient health outcomes. Is that right?
Yes, that's the primary orientation. I would say the efficiency gains are going to come no matter what. AI systems thrive on making things more efficient.
Now, what is efficient for a health insurance company, for an insurer, may not be efficient for you and I as patients. So, as [major health insurance companies] make things more efficient for themselves, that is going to lead to worse outcomes for us, potentially.
Lastly, I want to shift into the politics of all of this. Already, there's been some talk from the Trump administration about having government-backed plans using AI for decisions. Is that where we're headed?
It's going to get there, right? I mean, the Centers for Medicare and Medicaid Services are deploying AI at scale already, and it's a matter of time before it filters down to Medicare and Medicaid.
There's no holding this dam back. It's going to break. But I'm not necessarily worried about the dam breaking as much as I'm worried about how we channel the water. I'm a proponent for making sure that there are guardrails around AI. As full of potential as it is, you still need to channel that power toward good, otherwise you could end up with very unintended consequences.
Gabriella Paul covers the stories of people living paycheck to paycheck in the greater Tampa Bay region for WUSF. Here’s how you can share your story with her.
