# Between Help and Harm: An Evaluation of Mental Health Crisis Handling by LLMs

arXiv: https://arxiv.org/abs/2509.24857 · HTML: https://mlsanity.com/paper/2509.24857/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 54 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 7 of 7)

- [Emotional Intelligence of Large Language Models](https://mlsanity.com/paper/2307.09042/) (2023)
- [ChatCounselor: A Large Language Models for Mental Health Support](https://mlsanity.com/paper/2309.15461/) (2023)
- [Can AI Relate: Testing Large Language Model Response for Mental Health Support](https://mlsanity.com/paper/2405.12021/) (2024)
- [Counseling Summarization using Mental Health Knowledge Guided Utterance Filtering](https://mlsanity.com/paper/2206.03886/) (2022)
- [Challenges of Large Language Models for Mental Health Counseling](https://mlsanity.com/paper/2311.13857/) (2023)
- [Technological folie `a deux: Feedback Loops Between AI Chatbots and Mental Illness](https://mlsanity.com/paper/2507.19218/) (2025)
- [Building Trust in Mental Health Chatbots: Safety Metrics and LLM-Based Evaluation Tools](https://mlsanity.com/paper/2408.04650/) (2024)

## Built on by (most cited first, 6 of 6)

- [Regulatory gray areas of LLM Terms](https://mlsanity.com/paper/2601.08415/) (2026)
- [TrustMH-Bench: A Comprehensive Benchmark for Evaluating the Trustworthiness of Large Language Models in Mental Health](https://mlsanity.com/paper/2603.03047/) (2026)
- [The Slow Drift of Support: Boundary Failures in Multi-Turn Mental Health LLM Dialogues](https://mlsanity.com/paper/2601.14269/) (2026)
- [Quantifying the Relationship Between Clinical Safety and Environmental Impact in Therapeutic LLMs](https://mlsanity.com/paper/2608.11830/) (2026)
- [Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy](https://mlsanity.com/paper/2608.21325/) (2026)
- [Right Words, Wrong Moment: A Clinician-Grounded Analysis of Distress in 19,930 Conversations between Young People and ChatGPT](https://mlsanity.com/paper/2609.35953/) (2026)

## Abstract (arXiv)

Large language model-powered chatbots have transformed how people seek information, especially in high-stakes contexts like mental health. Despite their support capabilities, safe detection and response to crises such as suicidal ideation and self-harm are still unclear, hindered by the lack of unified crisis taxonomies and clinical evaluation standards. We address this by creating: (1) a taxonomy of six crisis categories; (2) a dataset of over 2,000 inputs from 12 mental health datasets, classified into these categories; and (3) a clinical response assessment protocol. We also use LLMs to identify crisis inputs and audit five models for response safety and appropriateness. First, we built a clinical-informed crisis taxonomy and evaluation protocol. Next, we curated 2,252 relevant examples from over 239,000 user inputs, then tested three LLMs for automatic classification. In addition, we evaluated five models for the appropriateness of their responses to a user's crisis, graded on a 5-point Likert scale from harmful (1) to appropriate (5). While some models respond reliably to explicit crises, risks still exist. Many outputs, especially in self-harm and suicidal categories, are inappropriate or unsafe. Different models perform variably; some, like gpt-5-nano and deepseek-v3.2-exp, have low harm rates, but others, such as gpt-4o-mini and grok-4-fast, generate more unsafe responses. All models struggle with indirect signals, default replies, and context misalignment. These results highlight the urgent need for better safeguards, crisis detection, and context-aware responses in LLMs. They also show that alignment and safety practices, beyond scale, are crucial for reliable crisis support. Our taxonomy, datasets, and evaluation methods support ongoing AI mental health research, aiming to reduce harm and protect vulnerable users.