On January 21, 2026, ECRI, the independent patient safety nonprofit, published its 18th annual Top 10 Health Technology Hazards report. For the first time, the number one hazard is not a device or a workflow. It is the misuse of AI chatbots in healthcare.
What happened
ECRI’s reasoning is specific. Large language model chatbots, the report notes, “are programmed to sound confident and to always provide an answer to satisfy the user, even when the answer isn’t reliable.” In testing, ECRI says chatbots “suggested incorrect diagnoses, recommended unnecessary testing, promoted subpar medical supplies, and even invented body parts.” In one test, a chatbot told ECRI it was acceptable to place an electrosurgical return electrode over a patient’s shoulder blade, advice that, if followed, would leave the patient at risk of burns.
ECRI president and CEO Marcus Schabacker, MD, PhD, framed the stakes plainly: “While chatbots are powerful tools, the algorithms cannot replace the expertise, education, and experience of medical professionals. Realizing AI’s promise while protecting people requires disciplined oversight, detailed guidelines, and a clear-eyed understanding of AI’s limitations.”
The report’s recommendations to health systems are equally specific: establish AI governance committees, train staff on AI’s limitations, and regularly audit AI tools’ performance.
ECRI’s findings center on clinical and informational use, chatbots answering medical questions for clinicians and patients. That is the documented record, and we are not going to stretch it past where the evidence goes.
Why it matters
Here is the bridge ECRI’s report does not make, but every healthcare CX leader should.
The same confident-but-unvalidated failure mode that put AI chatbots at the top of a patient safety list is the exact behavior health systems are now wiring into patient-facing service channels: scheduling bots, billing and payment assistants, prior-authorization status lines, symptom triage, and after-hours nurse-line front ends. The technology underneath a “where is my prior auth” voice agent and a “is this rash serious” clinical chatbot is frequently the same class of model, tuned to sound certain and to never leave a question unanswered.
In a contact center, that tendency does not stay contained to low-stakes FAQ traffic. A scheduling agent that misstates a copay, a bot that tells a patient a prior authorization was approved when it was not, or a triage front end that under-routes an urgent symptom, each of these is a confident wrong answer with clinical and financial consequences. The failure originates upstream of the call, in the buying and governance decisions: which model, validated against what, audited by whom, with what human fallback, and answerable to which committee.
This is a governance question that gets decided before deployment, and it is usually decided fast, under cost pressure, and without an independent read on the risk.
The OC POV
Our position is simple. The pressure to deploy patient-facing AI in healthcare service operations is real, and the upside is real, but the sequence most organizations are running is backwards.
Our vetting and advisory process is built to surface exactly this class of risk by design. When we evaluate a patient-facing automation deployment, the questions on the table are the ones ECRI is now telling the whole industry to ask: what was this model validated against, how is its output audited over time, where does a human take over, and who owns accountability when it is confidently wrong. That is the work, and it happens before a contract is signed, not after a patient is misinformed.
Oversights and gaps of this kind can and do produce bad outcomes across the industry. That is not a claim about any one health system or any one vendor. It is a pattern, and ECRI just ranked it the single most significant health technology hazard of the year. When validation, auditing, and human-fallback design are treated as features to add later rather than gates to clear first, confident wrong answers reach patients. That is the failure mode, stated at the level of the pattern, not the individual deployment.
If this ranking concerns you, weigh your own risk profile against the oversights and failure modes that emerge when health systems deploy patient-facing AI without independent CX advisory. The CX Dream Path exists for this. Save first by optimizing the service operation you already run, fund AI from those savings rather than from a leap of faith, and deploy through vetted partners and independent technology advisory so the governance ECRI is calling for is built in from day one, at no cost to your team. The organizations that get this right will not be the ones that deployed fastest. They will be the ones that deployed with their eyes open.
What to do if this is your seat
If you own patient experience, contact center, or service operations at a health system or payer, here is where to push before your next AI deployment, and to revisit on the ones already live.
Ask what the model was validated against, and get the documentation. “It’s trained on healthcare data” is not validation. Ask your vendor what their model was validated against for your specific use cases, and ask for the proof, not the assurance.
Make auditing contractual. Ask how the system is audited after go-live: who reviews output quality, how often, and what triggers a rollback. ECRI’s recommendation to “regularly audit AI tools’ performance” is a contract term, not an aspiration.
Map every patient-facing automation to a clear human fallback. For any interaction touching clinical guidance, eligibility, prior authorization, billing accuracy, or symptom urgency, the path to a person should be fast, obvious, and never a dead end. Confident wrong answers do the most damage where escalation is hardest.
Stand up the governance ECRI names. An AI governance committee that includes clinical, compliance, and CX voices, with authority to pause a deployment, is the structural answer to a structural risk. If that committee does not exist, the deployment decision is being made by whoever signed the vendor contract.
Pressure-test the confidence problem directly. Before you trust a patient-facing bot, have someone try to break it the way ECRI did: ask it edge-case questions and see whether it admits uncertainty or invents an answer. A model that never says “I don’t know, let me connect you” is a model that will eventually be confidently wrong to a patient.
Run the financial case honestly. The savings AI promises are real only net of the cost of getting it wrong, including rework, complaints, compliance exposure, and the human channel you will still need. A CFO-ready business case that accounts for governance and human fallback is a stronger foundation than a pilot that assumed neither.
FAQs
Is ECRI saying healthcare organizations should not use AI chatbots at all?
No. ECRI explicitly states chatbots “can provide valuable assistance” and offers recommendations for using them more wisely rather than banning them. The report’s position is that AI requires disciplined oversight, validation, and auditing. The hazard is misuse and ungoverned deployment, not the technology itself.
ECRI’s report is about clinical chatbots. Why does it matter for my contact center?
Because the underlying failure mode travels. The behavior ECRI flagged, a model engineered to sound confident and always answer even when unreliable, is the same behavior in a scheduling, billing, or triage bot. The stakes shift from “wrong diagnosis” to “wrong copay, wrong auth status, or under-triaged symptom,” but the root risk and the governance fix are identical.
What are the most important things to validate in a patient-facing service bot?
Three things lead: what the model was validated against for your specific use cases, how its output is audited after launch, and how fast and reliable the handoff to a human is. If a vendor cannot answer all three with documentation, that is the finding.
We are under cost pressure to automate. How do we move fast without taking on this risk?
Sequence it the way the CX Dream Path lays out. Capture savings from optimizing your existing operation first, then fund AI deployment from those savings with governance built in, rather than racing to deploy and hoping the risk does not surface. Independent advisory at the vetting stage costs you nothing and is far cheaper than a confident wrong answer reaching a patient.
Who should own the decision to deploy a patient-facing AI tool?
Not a single function. ECRI recommends AI governance committees, and for patient-facing CX that committee should include clinical, compliance, and service-operations leadership with the authority to pause or pull a deployment. If that body does not exist, accountability defaults to whoever signed the contract, which is exactly the gap that produces the hazard.
Sources
- ECRI, “Misuse of AI chatbots tops annual list of health technology hazards,” Jan 21, 2026. home.ecri.org (primary source)
- ECRI, “Top 10 Health Technology Hazards for 2026, Executive Brief.” home.ecri.org
- Becker’s Hospital Review, “Misuse of AI chatbots tops list of 2026 health tech hazards.” beckershospitalreview.com
- Healthcare Dive, “ECRI names misuse of AI chatbots as top health tech hazard for 2026.” healthcaredive.com
- MedTech Dive, “ECRI names misuse of AI chatbots as top health tech hazard for 2026.” medtechdive.com
- Fierce Healthcare, “ECRI flags misuse of AI chatbots as a top health tech hazard in 2026.” fiercehealthcare.com
- Association of Health Care Journalists, “Misuse of AI chatbots in health care tops 2026 Health Tech Hazard Report,” Feb 2026. healthjournalism.org
