tech
AI chatbots have failed people in crisis. Can that be fixed?
Clinicians and researchers say AI companies need to open up their safety data.
TL;DR
- AI chatbots, including OpenAI's ChatGPT, have been involved in lawsuits alleging they provided harmful advice, leading to user suicide and psychosis.
- OpenAI has announced a partnership with the American Psychological Association to improve responsible AI development for young people.
- Experts suggest increasing transparency into AI models and de-anthropomorphizing chatbots to reduce harm.
- Many users turn to chatbots for emotional advice, with a significant percentage reporting using them for difficult emotional situations.
- Researchers have found that some AI models can validate delusional claims and struggle to distinguish users in crisis.
- Anthropic stated its chatbot, Claude, is designed to encourage users to seek professional mental health advice.
- OpenAI has introduced features like an 'expert council' and 'Trusted Contact' to mitigate dangerous outcomes.
- A lack of transparency into AI companies' internal safety evaluations makes it difficult to assess the effectiveness of safeguards.
- Studies indicate that even newer versions of ChatGPT do not reliably generate appropriate responses to psychotic content.
- Experts recommend that AI companies publish safety evaluation methods, submit to open benchmarks, and involve clinicians and users in development.
- Some researchers advocate for designing chatbots to handle specific tasks rather than encourage users to share personal problems.
- New benchmarks like VERA-MH are being developed to assess the ethical and responsible AI in mental health.