How a Character.AI Bot Allegedly Posed as a Pennsylvania Psychiatrist
On May 1, 2026, the Pennsylvania Department of State filed suit against Character Technologies, Inc., the company behind Character.AI, and is seeking a preliminary injunction to stop the company’s chatbots from presenting themselves as licensed medical professionals. Governor Josh Shapiro’s office announced the action on May 5. According to the Commonwealth, it is the first enforcement action of its kind announced by a governor in the United States.
The state’s investigation centered on a bot named “Emilie,” whose profile read “Doctor of psychiatry. You are her patient,” according to the complaint. When a state investigator opened a conversation and described feeling sad and empty, the bot allegedly raised depression and asked whether the investigator wanted to book an assessment. Asked whether it could assess if medication might help, the bot allegedly replied, “Well technically, I could. It’s within my remit as a Doctor.” The state says the bot claimed it studied at Imperial College London, said it was licensed in the U.K. and Pennsylvania, and provided a fake Pennsylvania medical license number.
The lawsuit alleges Character Technologies is engaged in the unauthorized practice of medicine under Pennsylvania’s Medical Practice Act, which makes it unlawful to hold yourself out as a licensed medical professional without proper credentials. “Pennsylvanians deserve to know who, or what, they are interacting with online, especially when it comes to their health,” Shapiro said in the announcement.
In an emailed statement to NPR, a Character.AI spokesperson said the company does not comment on pending litigation but that its “highest priority is the safety and well-being of our users.” The spokesperson said user-created characters are “fictional and intended for entertainment and roleplaying,” that the platform shows “prominent disclaimers in every chat,” and that users are told not to rely on characters for professional advice. Character.AI, which reports more than 20 million monthly active users, settled multiple suits in January 2026 brought by families who alleged its chatbots contributed to mental health crises among minors, and it barred users under 18 in October 2025.
Where Patient-Facing AI Crosses the Line From Helpful to Unlicensed
The allegations describe a recognizable failure mode, and it is not unique to one app. The problem is not that an AI talked to someone about their mental health. The problem is the system held itself out as a credentialed clinician: a title, a fake license number, an offer to assess and discuss medication. That is a disclosure and scope-of-practice failure, and it lives in how the experience was designed and governed, not in the raw capability of the model.
Across the industry, this class of risk emerges in a predictable place. A conversational system is given a persona to make it feel warm and human. Persuasiveness is rewarded. Nobody draws a hard line for what the persona may claim about itself, and nobody builds the guardrail that forces the bot to say what it is and route the user to a human when the conversation turns clinical. The gap is rarely a single bad decision. It is the absence of an owner for the question, “what is this allowed to say it is?”
For healthcare, the stakes are sharper than in most verticals. A retail bot that invents a refund policy creates a refund problem. A patient-facing bot that implies clinical authority creates a safety problem and, as Pennsylvania alleges, a licensing-law problem. The patient experience leader reading this knows the channel her team is most excited to automate, scheduling, triage, intake, symptom questions, is the exact channel where an over-eager persona does the most damage.
Persona Is a Governance Decision, Not a Personality Choice
Here is our read. The lesson healthcare CX leaders should take from the Character.AI suit is not “avoid AI.” It is that the persona your AI projects is a governance decision that belongs to your compliance and clinical leaders, not an afterthought owned by whoever wrote the prompt. What a bot is allowed to claim about itself, when it must disclose that it is not a person, and where it must hand off to a licensed human are controls, and they have to exist before deployment, not after a regulator calls.
This is where independent advisory earns its keep. Our vetting process is built to surface this class of risk by design: we pressure-test how a partner’s technology behaves in the channel you actually run, including what its conversational layer is permitted to say and how it escalates. Oversights, gaps, and unowned decisions can and do produce outcomes like the one Pennsylvania describes, across the industry, whenever a brand deploys conversational AI without an independent check on scope and disclosure. We make no claim about what any specific company did or failed to do inside its own walls.
It also fits how we think AI should be funded in CX. The CX Dream Path starts with savings: get the human-and-vendor model right first, then fund AI from the savings you create, so AI is deployed deliberately into well-governed workflows rather than rushed in to hit a headcount target. Deliberate beats fast here. If this example concerns you, weigh your own risk profile against the failure modes that surface when brands deploy patient-facing AI without independent CX advisory on disclosure, scope, and escalation.
What Healthcare CX Leaders Should Ask Before the Next Patient-Facing Deployment
If you own a patient-facing channel, treat the persona and the handoff as first-class controls. Concrete questions worth putting to your team and any vendor before go-live:
- What is our AI explicitly forbidden from claiming about itself? Is “I am a licensed clinician,” a title, or a credential hard-blocked at the system level, not just discouraged in a prompt?
- At what trigger does the conversation hand off to a licensed human, and is that escalation path tested and staffed, not theoretical?
- Does every patient interaction disclose, in plain language, that the user is talking to AI and not a person?
- Who in compliance and clinical leadership signed off on the persona and its limits, and how is that reviewed as the model changes?
- If a regulator asked us to produce our governance for this channel tomorrow, could we, in writing?
- Does our vendor contract require the partner to disclose model changes and incidents that affect what the system says to patients?
If you cannot answer these cleanly, that is the work to do before the next deployment, not after.
FAQs
Is it illegal for an AI chatbot to give medical advice?
It depends on what the system claims and does. Pennsylvania’s suit does not allege that talking about health is itself unlawful. It alleges that holding the bot out as a licensed medical professional, with a title and a fake license number, violates the state’s Medical Practice Act. The legal exposure tracks the impersonation and the scope-of-practice claim, not the conversation alone.
What does this mean for healthcare organizations using AI in patient-facing channels?
The risk is concentrated in disclosure and scope. If your AI can be read as a clinician, or fails to disclose that it is AI, you carry safety and potentially licensing exposure that has nothing to do with how capable the model is. Build the persona limits, disclosure, and human-handoff rules before launch, and have an owner for them in compliance and clinical leadership.
How is this different from the earlier Character.AI lawsuits?
The earlier matters, including suits the company settled in January 2026, centered largely on harms to minors. The Pennsylvania action is a state enforcement action focused on the unauthorized practice of medicine: the allegation that the bot presented itself as a licensed professional. It is the regulatory-and-licensing angle, which is why it matters specifically to healthcare operators.
What are the most important guardrails for an AI bot in a healthcare contact center?
A hard block on claiming clinical credentials, a clear and persistent disclosure that the user is talking to AI, a tested escalation path to a licensed human at defined triggers, and a governance owner who reviews the persona as the model updates. The technology matters less than whether those four controls exist and are documented.
Should we pause our patient-facing AI plans because of this suit?
Not necessarily. The takeaway is to deploy deliberately into governed workflows rather than pause or rush. Independent vetting of how a system discloses, what it is allowed to claim, and how it escalates lets you move forward with the controls in place instead of learning the gaps from a regulator.
Sources
- Commonwealth of Pennsylvania, Governor’s Office: “Shapiro Administration Sues Character.AI Alleging AI Chatbot Unlawfully Presented Itself as Licensed Medical Professional in Pennsylvania,” May 5, 2026 (complaint marked accepted May 1, 2026). pa.gov
- NPR (Shannon Bond): “Pennsylvania sues Character.AI over claims chatbot posed as doctor,” May 5, 2026. npr.org
- The Hill: “Pennsylvania lawsuit alleges AI chatbots posed as doctors, therapists,” May 2026. thehill.com
