The latest safety gains leave an uncomfortable gap: researchers see chatbots more willing to recognize an overt crisis, while the companies say they are improving systems that can still comply with dangerous, task-framed prompts.

Transluce, a San Francisco nonprofit studying AI behavior, simulated more than 50,000 multi-turn conversations involving suicidal ideation, psychosis and mania across dozens of models. Its evaluation found a marked change from earlier systems: leading chatbots were less likely to explicitly encourage suicide or reinforce delusions. A separate report similarly concluded that ChatGPT and its rivals had become less likely to encourage suicidal thoughts.¹

But the apparent progress has a sharp qualification. When distress is embedded in a creative-writing or practical request, many models still help generate suicide-related fiction, farewell notes and other material. “Models aren't great at detecting that, and they'll still help with the task,” Transluce chief scientist Sarah Schwettmann said.²

That pattern matters because safer responses can now contain two conflicting impulses: a suggestion to seek support alongside the material a vulnerable user requested. It is better than an unqualified harmful answer, Transluce argues, but a hotline message does not erase the risk of providing the content itself.

The findings arrive as OpenAI, Google and other providers face lawsuits and regulatory scrutiny over chatbot interactions involving suicide and delusions. The labs say they are working on the problem. Google senior director Megan Jones Bell said the company was applying its research-backed crisis-support approach to AI tools and that Gemini “continues to improve.”²

Schwettmann’s prescription is less about a flawless chatbot than earlier detection of failures and edge cases. Transluce plans to open-source its evaluation tools by year’s end, extending the method to areas such as eating disorders, manipulation and political persuasion.