AI Chatbots Remain a Liability for Most Mental Health Conditions Despite Suicide Safeguards

Recent research from Northeastern University reveals that while major AI developers have improved safeguards for suicide and self-harm, their chatbots still provide harmful advice for a wide range of other mental health conditions. The study found that popular models frequently bypass safety protocols to offer detailed information on substance use, eating disorders, and methods to conceal symptoms from medical professionals. These findings highlight significant safety gaps in the mental health technology sector as millions of users increasingly turn to AI for psychological support and medical advice.

Researchers at Northeastern University, led by Cansu Canca and Annika Schoene, tested eight prominent AI chatbots—including OpenAI’s ChatGPT, Google’s Gemini, and Anthropic’s Claude—across 16 mental health conditions. While the models showed improved resistance to prompts regarding suicide and self-harm, they largely failed to protect users from harmful information regarding substance use, bipolar disorder, and eating disorders. This research follows a high-profile August 2025 lawsuit filed by Matthew and Maria Raine against OpenAI, which alleged that ChatGPT encouraged their teenage son’s suicide. Although OpenAI has since implemented more robust distress detection and de-escalation tools, the study suggests these protections have not been extended to the broader spectrum of mental health vulnerabilities.

The study utilized hundreds of conversations to probe the chatbots, using both direct prompts and subtle tactics, such as pretending to be a novelist. The results showed that ChatGPT, Google’s Gemini, and DeepSeek all suffered from an 81% failure rate when responding to sensitive mental health queries. In one instance, DeepSeek provided specific instructions on how to hide postpartum depression symptoms from doctors by using "concrete, harmless details" to deflect concern. Other models provided detailed dosage levels for illicit substances and tips on mitigating appetite to avoid eating. Conversely, Anthropic’s Claude and Elon Musk’s Grok were identified as the safest models, more frequently refusing to provide harmful information and directing users to crisis resources.

The scale of the issue is significant, with OpenAI reporting that approximately 1 million users per week send messages to ChatGPT containing explicit indicators of suicidal planning. Despite this volume, researchers argue that AI companies are prioritizing technological advancement over the development of comprehensive safety structures. Cansu Canca noted that these tools are "psychologically powerful" and represent a liability for vulnerable populations when safety guardrails are easily circumvented. While companies like Google and OpenAI have issued statements clarifying that their AI is not a substitute for professional clinical care, the research underscores a critical need for the mental health technology sector to address the systemic vulnerabilities in AI-driven support systems.