Making mental health chatbots more culturally sensitive won’t necessarily make them safer

Demand for mental health care in Canada has risen sharply. In a health-care system already short of publicly funded specialists, Canadians often wait months for support. Many are filling this gap with general-purpose generative AI chatbots (like ChatGPT), presumably because these systems are free, immediate and available at all hours.

However, these chatbots carry assumptions about what distress is and how it should be described, and those assumptions may not fit everyone.

In 2024–25, there were roughly 94,000 referrals for community mental health counselling across the provinces and territories that report this data. Half of those people were seen within 30 days. One in 10 waited more than four months.

A 2026 survey of more than 1,200 licensed psychologists in the United States found that 77 per cent had patients who told them they were using AI for mental health support, and more than one-third had patients who considered their chatbot an additional provider. These tools have not been approved by Health Canada, and no Canadian regulation governs them as therapy.

The data also suggest that the people most likely to depend on chatbots for support are the ones least likely to be culturally understood by them: newcomers, people in rural and remote communities and anyone who cannot afford private therapy.

Chatbots are not culturally neutral

When a chatbot responds to someone in distress, it’s already committed to a view of what distress is and what helps. Its default assumes distress sits inside the individual, is made of thoughts and feelings, and improves when those thoughts are examined and changed.

That is a recognizable clinical tradition (broadly western and cognitive-behavioural), when an individualistic concept of the person has long been identified as culturally particular rather than universal. In general-purpose AI chatbots, this surfaces as a tendency toward the values of English-speaking populations in the Global North.

As a PhD candidate in cultural psychiatry, I know the discipline has long made this argument in the clinical context. Psychiatrist and anthropologist Arthur Kleinman described the “category fallacy” in 1977 after working with Chinese patients whose depression presented largely as bodily complaint (fatigue, dizziness, insomnia and pain rather than reported sadness or guilt).

Sociocultural anthropologist Mark Nichter’s 1981 work on idioms of distress demonstrated how suffering is voiced through whatever channels a community makes available. For the women he studied in southern India, that again meant feeling distress in the body instead of voicing it directly.

Distress expressed through the body, faith or obligation to family isn’t a confused version of psychological distress. A person may describe their suffering as a trial sent by God, a loss of faith, a spiritual affliction or a consequence of moral failing; another may locate it not in their own mood but in what they owe others — shame at failing an aging parent, guilt at not providing or the sense of having brought difficulty onto the family. It’s how a great deal of suffering is communicated, especially in a country as diverse as Canada.

AI chatbots handle this poorly. Studies testing whether chatbots pick up on cultural cues find they often struggle to identify a cultural pattern, especially when prompted in English. Recent research found that models express stigma toward people with schizophrenia and alcohol dependence, respond inappropriately in realistic therapy scenarios, and that newer and larger models did not improve.

This failure can be categorized as imposition, to borrow the term from the account of Madeleine Leininger, who developed the concept of transcultural nursing and of cultural imposition in clinical care. Imposition is when the system applies its own model of distress to someone that model does not serve. I describe this further in my article about the dilemma(s) associated with cultural adaptation and the use of AI chatbots for mental health.

Would a more culturally sensitive chatbot be safer?

The obvious response is to adapt these systems by training on more languages, using local idiom and building tools with and for specific communities. That is important, but it presupposes that cultural fit is the same thing as cultural safety. It’s not.

Some ways of understanding distress keep people inside it by ruling out the actions that might relieve it. For example, a person may believe their condition would shame their family if anyone were to find out, or that they are not unwell but weak and undisciplined. A chatbot with good cultural fit might reflect those beliefs back sympathetically, in the user’s own vocabulary. In psychotherapy this is called collusion. It occurs when a therapist goes along with a client’s account in a way that prevents anything from changing.

Chatbots are prone to collusion because of how they are built and trained (largely on human approval ratings). To the user, being understood and being agreed with feel much the same, so both earn approval, and the training reinforces both together. The system therefore has no way to tell recognition from endorsement.

That produces an awkward conclusion. The better a model’s cultural fit, the stronger its incentive to agree with what the user already believes, and the more convincing that agreement will be. This also applies to western accounts of distress — if you insist you’re not depressed and you’re just not working hard enough, a chatbot could readily agree with you.

The imposition side is already well-documented. The collusion side in its cultural form has not been measured, despite the underlying tendency being well established: sycophancy in language models has been demonstrated repeatedly, and 97 per cent of psychologists in the U.S. survey worry that chatbots might reinforce negative behaviours or delusional beliefs. Nothing in an approval-based signal separates a belief someone reached alone from one absorbed from their culture or community.

What would make these systems safer

If a chatbot is judged as a safe mental health tool because its users felt understood, we are using an instrument that cannot separate good care from agreeable care.

A more useful test looks at where the conversation ends — did the person end up more limited in what they feel able to do or further from help than they wanted? Those questions can be answered without deciding in advance whose understanding of distress is correct (otherwise calling something collusion is simply imposition by another name).

National regulation of these tools is being discussed, and many safety failures will likely be addressed. However, the failure described here will not look like a safety failure — it will only look like being understood.