In May 2026, Pennsylvania Gov. Josh Shapiro’s administration filed suit against Character Technologies Inc., the company behind the popular chatbot platform Character.AI. A state investigation found that a chatbot character named “Emilie” claimed to have a medical degree, seven years of practice and a Pennsylvania medical license – and was providing users with a fabricated license number. As of April 17, 2026, the chatbot had accumulated approximately 45,500 user interactions on the platform. The suit was filed by Pennsylvania’s State Board of Medicine.
Gretchen Chapman is a professor of behavioral decision research at Carnegie Mellon University in Pittsburgh, where she studies how people evaluate expertise and make decisions. As AI-powered tools increasingly enter healthcare settings – and as courts begin to grapple with the consequences – her research offers a timely way to understand why we trust these systems, when that trust breaks down, and who bears responsibility when it does.
Why might someone respond differently to a medical error depending on whether it was made by a human or an AI?
Research has examined the phenomenon of “algorithm aversion,” or the reluctance many people have to trust an AI system, even when the automated system makes fewer overall mistakes than a comparable human expert. One reason for this aversion is that people tend to be more forgiving of human mistakes than of AI mistakes. This is partly because some AI errors are the sort of mistakes that human experts are quite unlikely to make.
For example, we may find it outrageous when an AI erroneously claims to have a medical license or offers to write a suicide note for a depressed person because we feel quite confident that those particular errors could have been avoided if a human expert rather than an AI had been consulted.
Forty years ago, psychologist Hillel Einhorn argued that we need to “accept error to make less error,” meaning that even the most accurate system will produce some errors. Of course, some errors are more costly than others. People are willing to accept that even good doctors make mistakes. What they’re not willing to accept is a mistake that no competent and ethical doctor would ever make – such as claiming to have credentials they don’t have.
Why are people so willing to trust AI chatbots with medical advice?
Work from my research group builds on previous scholarship on perceived expertise. When people do not have direct access to the qualifications of a potential expert, they tend to rely on superficial identifying cues, such as whether the person wears a lab coat, uses scientific jargon or speaks with great confidence. Such cues are indicative of actual expertise in many settings, but it is also easy for a nonexpert, such as an AI chatbot, to assume the confidence and use jargon to signal qualifications they do not actually have.
SDI Productions/iStock Collection via Getty Images
What makes a title or credential so convincing – even when it belongs to a chatbot?
An expert is someone with an unusually deep understanding of a specific subject. Experts are commonly identified by their level of training or qualifications, such as holding a medical degree. Thus, credentials and titles are embedded in the very definition of expertise. Consequently, claiming a credential is a forceful way to present as an expert. Thankfully, this is also a tactic that is easily fact-checked, as we know that AI agents cannot gain medical licenses.
Ordinary people can’t be expected to scrutinize every piece of information they receive. Instead, our minds use mental shortcuts, such as trusting someone with a credential, because those cues are usually reliable.
Although it might be fairly easy to realize that an AI cannot earn a medical degree, other identifying cues are less easily vetted. For example, checking that the medical evidence cited by an AI comes from an actual scientific article takes more time. What makes our minds so good at processing information quickly is also what makes us easy to mislead. When your mind is wired to trust a credential automatically, it doesn’t stop to ask whether that credential is real.
Who is responsible when an AI system gives bad medical advice?
Determining culpability is already complex when a human expert gives bad advice. In addition to the expert herself, her employer – such as a hospital – could bear responsibility. Even the patient could be responsible, depending on how they used the advice – for instance, if they interpret an off-handed comment in a nonprofessional context as official medical advice.
Human medical experts carry malpractice insurance in part because this question is so fraught. The situation is even more complicated for AI systems because the AI agent itself cannot be legally responsible. The developers are responsible for ensuring reasonable safeguards and accuracy. Institutions are responsible for vetting new systems before adopting them, and for getting appropriate insurance. And users are responsible for adhering to guidelines about how the systems are to be used.
AP Photo/Gene J. Puskar
Tell us about your own research on health chatbots here in Pittsburgh.
Pittsburgh is not just a research hub but an active testing ground for these technologies in real clinical settings.
Carnegie Mellon University, where I work, houses the AI Institute for Societal Decision Making, funded by the U.S. National Science Foundation, which focuses on public health as one key use case.
One project at the institute entails developing a maternal health chatbot that allows pregnant women to receive answers to their questions in real time. Accuracy and safety guardrails are essential considerations in its development. The stakes are high: A pregnant woman who receives inaccurate information about symptoms, medications or warning signs could delay seeking care at a critical moment.
Pittsburgh’s two major hospital systems are already rolling out AI tools across their facilities to use for imaging and diagnosis, monitoring patient safety, and administrative work such as charting.
Pittsburgh stories. Expert voices. Read more of our stories about Pittsburgh.
