Healthcare AI failures, scored
Two algorithms that advised on care: a sepsis alert and a risk score that chose patients for extra care. Each is read against the same six safeguards as the rest of the casebook. Each case names who carried the errors and whether anyone stopped the system. The scores are drafts.
Every case on this page was established by a court, a statutory inquiry, a regulator, or a peer-reviewed external validation. The main casebook admits only cases established by a court, an inquiry, or a regulator. This page admits one more kind of record: a peer-reviewed study in which researchers outside the developer tested the system. Both cases here rest on such a study. Neither would be admitted to the main casebook, and neither is counted in its scores. No press coverage is cited; a fact that rested only on press was left out.
Every score on this page is this project's own draft reading of the record. It is not a finding by the study's authors, a court, or a regulator, and it is not a verdict on anyone. Where a draft score rested only on press, it is withheld and the case says so.
Neither system has a halt on the record. When the record was last checked, on October 8, 2026, no record was found that either was withdrawn.
Each case ends with what could not be confirmed. The six safeguards are the ones the main casebook scores, and the method defines them.
held in place and working · drifted in place but out of step with what it should track · failed absent, or its absence is what the record shows
The Epic Sepsis Model
A sepsis alert in use at hundreds of US hospitals. The study says it had not been adequately evaluated. When one hospital checked, the model found a third of sepsis cases and alerted on nearly a fifth of all hospitalizations.
- System
- A proprietary model that Epic Systems built into its electronic health record. It scores hospitalized patients for sepsis every 15 minutes and alerts clinicians above a set score.
- Where
- United States · Epic Systems (vendor); Michigan Medicine (the deployment studied)
- When
- December 2018 to October 2019, the period the study covers
- Scale
- 38,455 hospitalizations of 27,697 adults at Michigan Medicine. The model alerted on 6,971 hospitalizations (18%) and missed 1,709 of the 2,552 patients with sepsis (67%). The study reports the model in use at hundreds of US hospitals.
- Halt
- None on the record as of Oct 8, 2026. No record found that the model was withdrawn.
- Errors carried by
- Patients with sepsis the model did not flag (1,709 of 2,552 in the study) and the clinicians paged by alerts on 6,971 hospitalizations.
- Admitted on
- A peer-reviewed external validation, published in JAMA Internal Medicine on 21 June 2021. No court, statutory inquiry, or regulator has made a finding about the model.
- Scores
- This project's draft reading.
What happened
- Dec 2018 The study period begins at Michigan Medicine. The model scores each patient every 15 minutes.
- Oct 2019 The study period ends: 38,455 hospitalizations, 2,552 of them with sepsis.
- Jun 2021 JAMA Internal Medicine publishes the external validation. The model missed 67% of sepsis cases and alerted on 18% of hospitalizations.
The Epic Sepsis Model is a proprietary score built into Epic's electronic health record. It recalculates every 15 minutes, and alerts fire at or above a set score; the study used 6. Michigan Medicine is the University of Michigan's academic health system. Its researchers evaluated the model on 38,455 hospitalizations of 27,697 adults admitted between 6 December 2018 and 20 October 2019. Sepsis occurred in 2,552 of them.
JAMA Internal Medicine published the study on 21 June 2021. It found an area under the curve of 0.63. The authors call that substantially worse than the performance reported by the model's developer. At a score of 6 or higher, the model alerted on 6,971 hospitalizations (18% of the total) and still did not identify 1,709 of the patients with sepsis (67%). It identified 183 patients with sepsis (7%) who had not already received timely antibiotics. The authors wrote that the model's widespread adoption despite poor performance raises fundamental concerns about sepsis management nationally.
No court or regulator has made a finding about the model.
The six safeguards, draft scores
-
failed
Capability
What could the whole system actually do?
The deployment relied on a model that could find sepsis. At the hospital that checked, it found a third of the cases. Of the 2,552 patients with sepsis, it flagged 183 (7%) whom clinicians had not already treated with timely antibiotics. The capability on the label was not the capability in the ward.
-
drifted
What was it allowed to do, to whom, and until when?
A score above a threshold had the authority to page a clinician. The study says the model's ability to find sepsis had not been adequately evaluated despite its use at hundreds of hospitals. The record found shows no evaluation behind the threshold that decided who was paged.
-
failed
Evidence
What evidence said it should be allowed to do that?
The performance reported for the model came from its developer. The external check at one hospital found it substantially worse. Evaluation that only the supplier performs is not detection the operator holds.
-
held
Dependency
How hard had it become to switch it off or replace it?
The record shows no lock-in. Michigan Medicine could evaluate the model on its own records and report the threshold it used. That is the precondition for switching an alert off or changing it. The cost the record shows is alert burden, not dependence.
-
failed
Standing
Could the people it decided about push back, and did anyone have to answer?
The record found shows no way for a patient the model missed to know a score existed, and no route by which the clinicians absorbing alerts on 6,971 hospitalizations could oblige the vendor to answer.
-
Not scored
Correction
Could anyone still step in, and how fast?
The draft reading of this safeguard rested on press reports, which this page does not cite. No record found that the model was withdrawn.
After the failure: did the institution change?
Not scored
The draft reading rested on press reports, which this page does not cite. No record shows hospitals that had already deployed the model re-validating it.
The record that was not there
An evaluation independent of the vendor, at the hospital where the alerts fire, before they go live. STD-08 §4.6 refuses to count detection that the supplier alone performs, and STD-06 §2.2 requires each threshold to be declared before the test runs.
What could not be confirmed
- The developer's reported performance is not stated here, though the study cites figures from Epic's documentation. This page says only what the abstract says: the model performed substantially worse.
- No record found of the model being withdrawn.
- No FDA action or finding specific to this model was found.
- The vendor's response to the study, and what happened to the model after it, were found only in press. They are left out, and the two readings that rested on them, correction and what the institution changed afterward, are not scored.
Avoid this failure in your system
1. Run the diagnostic
Takes one workflow and asks what evidence each automated action rests on and who produced that evidence.
2. Apply the standard
STD-06: Human Impact Safety Case
Requires each test's threshold to be declared before the test runs, with the action that follows a breach.
3. Read why it recurs
Why do a system's capability and track record never give it authority on their own?
Sources
Optum's cost-as-proxy risk score
A risk score that did exactly what it was built to do: predict cost. It was used for something else: finding the sickest patients. Unequal access to care made cost a worse measure of need for Black patients. The manufacturer confirmed the finding on 3.7 million patients.
- System
- A commercial risk score that health systems and insurers use to pick patients for high-risk care management. It predicts future health care cost and uses that as a stand-in for health need.
- Where
- United States · Optum (UnitedHealth Group), named by New York regulators as the maker of Impact Pro
- When
- 2013 to 2015, the years the study covers
- Scale
- At the same risk score, Black patients were considerably sicker than White patients. Removing the disparity would raise the share of Black patients identified for extra help from 17.7% to 46.5%. By industry estimates, tools of this kind are applied to about 200 million people in the US each year.
- Halt
- None on the record as of Oct 8, 2026. No record found that the score was withdrawn or that New York's inquiry reached a finding. The authors reported an unpaid collaboration with the manufacturer on a better label; no record found that a revised version was deployed.
- Errors carried by
- Black patients, who at the same level of illness generated $1,801 less in health care costs a year. They scored as healthier and were selected less often for extra care.
- Admitted on
- A peer-reviewed study, published in Science on 25 October 2019, in which researchers outside the manufacturer tested the score on one hospital's patients. New York's regulators demanded proof or a stop but made no finding on the record.
- Scores
- This project's draft reading.
What happened
- 2013–15 The patients the study follows are enrolled in risk-based contracts at one academic hospital and scored by the algorithm.
- Oct 2019 Science publishes the study. At the same risk score, Black patients are considerably sicker. The manufacturer has replicated the result on 3,695,943 patients.
- Oct 2019 New York's financial and health regulators ask UnitedHealth Group to show Impact Pro is not discriminatory, or stop using it.
Obermeyer, Powers, Vogeli, and Mullainathan studied primary care patients enrolled in risk-based contracts at one large academic hospital from 2013 to 2015. The hospital used a commercial algorithm to choose patients for care management programs: those above the 97th percentile of its score were automatically identified for enrollment. The algorithm predicted health care cost. Unequal access to care means less is spent on Black patients. At a given level of illness, they generated $1,801 less in costs a year, and so scored lower. At the same risk score, Black patients were considerably sicker. Removing the disparity would raise the share of Black patients identified for extra help from 17.7% to 46.5%. Science published the study on 25 October 2019.
Before publication, the authors took their results to the manufacturer. It replicated the analysis on its national dataset of 3,695,943 commercially insured patients and found Black patients had 48,772 more active chronic conditions than White patients at the same risk score. Changing the label from cost alone to an index of health and cost cut that to 7,758, an 84% reduction in bias. The authors described an ongoing unpaid collaboration to build a better predictor. The paper does not name the manufacturer.
On 25 October 2019, New York's Department of Financial Services and Department of Health wrote to UnitedHealth Group about Optum's Impact Pro. They asked it to investigate, show the algorithm was not racially discriminatory, or stop using it.
The six safeguards, draft scores
-
held
Capability
What could the whole system actually do?
The model predicted cost well. Measured as a cost predictor, it worked. Capability is not where this case failed.
-
drifted
What was it allowed to do, to whom, and until when?
A score built to predict spending was given the authority to decide who got extra care: above the 97th percentile, enrollment was automatic. The threshold traded one kind of error for another, and nothing on the record named which, or for whom.
-
failed
Evidence
What evidence said it should be allowed to do that?
Cost was the evidence; need was the decision. Black patients generated $1,801 less in costs a year at the same level of illness, so the evidence understated their need. By ordinary accuracy measures the proxy looked sound. That is why the gap went unseen.
-
drifted
Dependency
How hard had it become to switch it off or replace it?
The study says large health systems and payers rely on the algorithm, and tools like it reach about 200 million people a year. The fix the authors tested runs through the manufacturer's next round of development.
-
failed
Standing
Could the people it decided about push back, and did anyone have to answer?
For patients passed over for care management, the record found shows no notice that a score had been used and no route to question it. The disparity was found by outside researchers with access to the data, not by any challenge from the people it fell on.
-
drifted
Correction
Could anyone still step in, and how fast?
The manufacturer replicated the result and tested a new label that cut the bias by 84%. A regulator demanded proof or a stop. Neither has a recorded end: no record found shows a deployed fix or an outcome of the inquiry.
After the failure: did the institution change?
Partly changed
The manufacturer accepted the finding and began work on the label that produced the bias. That work reaches the rule and not just the cases. The record stops there. No record shows whether the revised score is in use or what happened to patients scored by the old one.
The record that was not there
Error rates by population, computed on a declared cadence from records the operator already holds (STD-06 §2.4). The disparity was in the data the score ran on. It took outside researchers to compute it.
What could not be confirmed
- The outcome of New York's inquiry was not found.
- No record found that a revised version of the score was deployed.
- The paper does not name the manufacturer; the attribution to Optum's Impact Pro rests on New York's letter.
- The manufacturer's public response to the study was found only in press, and is left out.
Avoid this failure in your system
1. Run the diagnostic
Takes one workflow and asks what evidence each decision rests on and who bears the errors when that evidence is wrong.
2. Apply the standard
STD-06: Human Impact Safety Case
Requires error rates measured per affected population from the operator's own records, with a materially worse rate for any group treated as a breach.
3. Read why it recurs
Why does judging an institution by its outcomes hide who carried the burden of producing them?
Sources
- Peer-reviewed study Obermeyer Z, Powers B, Vogeli C, Mullainathan S, Dissecting racial bias in an algorithm used to manage the health of populations, Science 366(6464):447–453 · Oct 25, 2019
- Regulator New York State Department of Financial Services and Department of Health, joint letter to UnitedHealth Group on Impact Pro · Oct 25, 2019
Cite this page · APA, MLA, Chicago, BibTeX, RIS
APA
Ethotechnics Institute. (2026). Casebook: healthcare AI failures, scored. Ethotechnics Institute. https://ethotechnics.org/casebook/healthcare-ai
MLA
Ethotechnics Institute. "Casebook: healthcare AI failures, scored." Ethotechnics Institute, 2026, https://ethotechnics.org/casebook/healthcare-ai.
Chicago
Ethotechnics Institute. "Casebook: healthcare AI failures, scored." Ethotechnics Institute. Nov 10, 2026. https://ethotechnics.org/casebook/healthcare-ai.
BibTeX
@misc{ethotechnics_casebook_healthcare_ai,
title={Casebook: healthcare AI failures, scored},
author={Ethotechnics Institute},
year={2026},
howpublished={Ethotechnics Institute},
url={https://ethotechnics.org/casebook/healthcare-ai}
}
RIS
TY - WEB TI - Casebook: healthcare AI failures, scored AU - Ethotechnics Institute PY - 2026 UR - https://ethotechnics.org/casebook/healthcare-ai ER -