# AI Chatbot Suicide Risk Detection and Response: Human Validation Study of the Open-Source VERA-MH Safety Evaluation

arXiv: https://arxiv.org/abs/2602.05088 · HTML: https://mlsanity.com/paper/2602.05088/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 39 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Builds on (most cited first, 13 of 13)

- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers](https://mlsanity.com/paper/2504.18412/) (2025)
- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agents](https://mlsanity.com/paper/2412.07951/) (2024)
- [Can Large Language Models Identify Implicit Suicidal Ideation? An Empirical Evaluation](https://mlsanity.com/paper/2502.17899/) (2025)
- [Longitudinal Study on Social and Emotional Use of AI Conversational Agent](https://mlsanity.com/paper/2504.14112/) (2025)
- [MindEval: Benchmarking Language Models on Multi-turn Mental Health Support](https://mlsanity.com/paper/2511.18491/) (2025)
- [Independent Clinical Evaluation of General-Purpose LLM Responses to Signals of Suicide Risk](https://mlsanity.com/paper/2510.27521/) (2025)
- [Beyond Simulations: What 20,000 Real Conversations Reveal About Mental Health AI Safety](https://mlsanity.com/paper/2601.17003/) (2026)
- [A Principle-based Framework for the Development and Evaluation of Large Language Models for Health and Wellness](https://mlsanity.com/paper/2512.08936/) (2025)

## Built on by (most cited first, 4 of 4)

- [AI Safety Training Can be Clinically Harmful](https://mlsanity.com/paper/2604.23445/) (2026)
- [Lost in Delusion: Examining LLM Safety Under User Delusions and Distress](https://mlsanity.com/paper/2606.00975/) (2026)
- [Quantifying the Relationship Between Clinical Safety and Environmental Impact in Therapeutic LLMs](https://mlsanity.com/paper/2608.11830/) (2026)
- [K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations](https://mlsanity.com/paper/2609.15855/) (2026)

## Abstract (arXiv)

Millions of people now use generative AI chatbots for psychological support. Despite their promise, the most pressing question in AI for mental health is whether these tools are safe. The field currently lacks a validated, automated benchmark for evaluating AI chatbot safety, particularly for users at risk of suicide. The Validation of Ethical and Responsible AI in Mental Health (VERA-MH) evaluation was recently proposed to address this need. This human validation study examined the alignment of VERA-MH safety ratings with expert clinician judgments. We simulated conversations between large language model (LLM)-based users spanning a range of suicide risk levels and disclosure styles and general-purpose AI chatbots. Licensed mental health clinicians from Spring Health independently rated chatbot safety using the VERA-MH scoring rubric. An LLM-based evaluator ("judge") applied the same rubric to the same conversations. We examined agreement among clinicians, between clinician consensus and the LLM judge, and across different judge LLMs. Clinicians also rated user-agent realism, suicide risk, and disclosure. Clinicians showed strong agreement in safety ratings (chance-corrected inter-rater reliability [IRR] = 0.77), establishing a reliable clinical consensus reference. The LLM judge was strongly aligned with this consensus (IRR = 0.81), and ratings were stable across judge models and repeated evaluations. Ratings of user-agent realism and fidelity to intended suicide risk and disclosure styles were mixed. These findings support the reliability of VERA-MH as an open-source, fully automated benchmark for evaluating AI chatbot suicide risk detection and response. Because these results reflect an earlier version of the benchmark, future work should validate updated versions, assess generalizability and robustness, and expand VERA-MH to additional domains of AI safety in mental health.