If you want to see where clinical AI actually breaks, skip the vendor deck. Watch season two, episode six, of The Pitt.
Dr. Baran Al-Hashimi (Sepideh Moafi) rolls out an AI tool that writes the charts. It cuts documentation time by eighty percent. More time at the bedside. More time at home. Who says no to that?
Then the AI invents a surgery the patient never had. The resident says there’s no surgical history. Now no one can admit the patient without a surgical consult, a delay the tool created out of nothing.
Al-Hashimi defends it. Two percent error is still better than dictation. She’s right about the number. But Dr. Robby (Noah Wyle) names what the number hides: “It’ll make us more efficient – but hospitals will expect us to treat more patients without extra pay.”
That’s the whole fight in one exchange. And it has almost nothing to do with how smart the AI is.
The wall isn’t intelligence.
Most of the conversation about clinical AI is about accuracy. Can the model read the scan or draft the note? Can it beat the resident on a benchmark?
I believe that’s the wrong question. Hospitals are businesses. They care for people, and they survive on two things: economics and liability. A tool can be brilliant and still never make it past the door, because the hospital leaders I talk to aren’t asking whether it’s smart. They’re asking what happens when it’s wrong, and who gets sued.
That second question is the one that will decide how fast clinical AI actually arrives. And right now, the honest answer scares them.
Detection creates a duty.
Here’s the part the technology pitches skip.
The moment an AI tool flags something, the hospital owns it. If the model surfaces a finding and no one acts on it, that finding becomes a documented omission. It sits in the record, with a timestamp, waiting for a lawyer to find it.
Doctors already have a word for the way they practice around this fear. Defensive medicine. You order the extra test, you write the careful note, you cover yourself, not always because the patient needs it but because the lawsuit might come. A widely cited Harvard study in Health Affairs put the cost at about $46 billion a year, and that was long before AI entered the room.
AI doesn’t relieve that pressure. It industrializes it. Every flag is a new thing you’re now on the hook for. Every AI-generated note you sign is a note you’ve certified as true. When the tool in The Pitt invents an appendectomy, the doctor who signs that chart owns the lie. Two percent error sounds small until you’re the one holding the two percent in front of a jury. Doctors have a name for the way it looks from there. The retrospectoscope. In hindsight, every decision looks obvious.
So put yourself in the seat of a general counsel or a chief medical officer. You’re being asked to deploy a tool that creates a new, searchable trail of everything your clinicians saw and didn’t act on, in a format a plaintiff’s attorney would love. You don’t need the tool to be bad for that to worry you. You only need it to be wrong occasionally, and findable always.
That is the liability wall. And I believe it is going to stop a lot of hospitals from implementing clinical AI, no matter how good the demo looks. Not because the technology fails. Because the risk of getting sued is real and the reward is still unclear.
There’s a trap underneath the wall.
Now here’s the harder problem, and it’s the one almost no one is naming.
For clinical AI to get good enough to lower that risk, it has to learn how doctors actually decide. Not the order that got placed but the thinking behind it. It must understand why an experienced physician distrusted a lab value that the chart couldn’t explain, why she watched one finding and chased another. Or about why she stopped the drug instead of the dose.
Doctors call this the art of medicine, or clinical gestalt. It’s the judgment that runs ahead of the guideline. It’s also the exact thing AI needs and can’t get, because the medical record was never built to hold it.
The record captures the decision. It does not capture the deliberation. It logs that she ordered the test, not the three diagnoses she considered and ruled out. And the reason it works that way is the same liability we just walked through. Recorded uncertainty is discoverable. The safest thing a doctor can do, legally, is keep the reasoning in her head.
So look at the loop. AI needs the reasoning to become trustworthy. The reasoning lives in the doctor’s head. Writing it down creates liability. So it stays in her head and the AI never learns it. That means the AI stays stuck at the level that creates more liability, not less.
The thing that would make clinical AI safe enough to deploy is the one thing the system is built, legally and structurally, to throw away.
Here’s what this means if you’re building or buying.
I’m not arguing against clinical AI. I’m arguing that the hard part was never the model.
If you’re buying, stop evaluating accuracy on its own. Ask the downstream questions. When this tool is wrong, who carries it? When it finds something real, who works it up, and who pays for that work. If your vendor can’t answer those, you’re not buying a capability. You’re buying risk with a nice interface.
If you’re building, stop selling accuracy. Sell what happens after the flag. Design for the doctor’s judgment to stay in the loop, because that judgment is both the thing that manages your liability and the thing your model most needs to learn from. The product that wins won’t be the one that finds the most. It’ll be the one a general counsel can sign off on.
Back to The Pitt
The show got something right that most of the industry keeps getting wrong. The fight on that ER floor wasn’t about whether the AI was smart. It was about who pays, who’s liable, and who’s left holding the chart when the machine is wrong.
Clinical AI won’t be won by the model that finds the most disease. It’ll be won by the system that figures out who answers for what it finds, and keeps the people who know which findings matter.
That’s not a technology problem. It’s a business problem wearing a lab coat.