# MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Models

arXiv: https://arxiv.org/abs/2604.17730 · HTML: https://mlsanity.com/paper/2604.17730/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 71 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Builds on (most cited first, 30 of 37)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation](https://mlsanity.com/paper/2310.17389/) (2023)
- [X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents](https://mlsanity.com/paper/2504.13203/) (2025)
- [CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility](https://mlsanity.com/paper/2307.09705/) (2023)
- [SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety](https://mlsanity.com/paper/2404.05399/) (2024)
- [Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data](https://mlsanity.com/paper/2307.14385/) (2023)
- [The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships](https://mlsanity.com/paper/2410.20130/) (2024)
- [Large Language Models Assume People are More Rational than We Really are](https://mlsanity.com/paper/2406.17055/) (2024)
- [Towards Human-centered Proactive Conversational Agents](https://mlsanity.com/paper/2404.12670/) (2024)
- [Knowledge-enhanced Mixed-initiative Dialogue System for Emotional Support Conversations](https://mlsanity.com/paper/2305.10172/) (2023)
- [MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance](https://mlsanity.com/paper/2503.13509/) (2025)
- [LLM Harmony: Multi-Agent Communication for Problem Solving](https://mlsanity.com/paper/2401.01312/) (2024)
- [GraphEval: A Knowledge-Graph Based LLM Hallucination Evaluation Framework](https://mlsanity.com/paper/2407.10793/) (2024)
- [ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks](https://mlsanity.com/paper/2310.12516/) (2023)
- [The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs](https://mlsanity.com/paper/2506.11094/) (2025)
- [From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agents](https://mlsanity.com/paper/2412.07951/) (2024)
- [From Personas to Talks: Revisiting the Impact of Personas on LLM-Synthesized Emotional Support Conversations](https://mlsanity.com/paper/2502.11451/) (2025)
- [Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMs](https://mlsanity.com/paper/2502.20968/) (2025)
- [Do Large Language Models Rank Fairly? An Empirical Study on the Fairness of LLMs as Rankers](https://mlsanity.com/paper/2404.03192/) (2024)
- [Chain of Strategy Optimization Makes Large Language Models Better Emotional Supporter](https://mlsanity.com/paper/2503.05362/) (2025)
- [LifeTox: Unveiling Implicit Toxicity in Life Advice](https://mlsanity.com/paper/2311.09585/) (2023)
- [Consistent Client Simulation for Motivational Interviewing-based Counseling](https://mlsanity.com/paper/2502.02802/) (2025)
- [CAMI: A Counselor Agent Supporting Motivational Interviewing through State Inference and Topic Exploration](https://mlsanity.com/paper/2502.02807/) (2025)

## Built on by (most cited first, 3 of 3)

- [Mental Health AI Safety Claims Must Preserve Temporal Evidence](https://mlsanity.com/paper/2605.08827/) (2026)
- [EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbots](https://mlsanity.com/paper/2606.30256/) (2026)
- [PeerPen: AI-Assisted Writing for Online Mental Health Peer Support](https://mlsanity.com/paper/2609.14886/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly explored as scalable tools for mental health counseling, yet evaluating their safety remains challenging due to the interactional and context-dependent nature of clinical harm. Existing evaluation frameworks predominantly assess isolated responses using coarse-grained taxonomies or static datasets, limiting their ability to diagnose how harms emerge and accumulate over multi-turn counseling interactions. In this work, we introduce R-MHSafe, a role-aware mental health safety taxonomy that characterizes clinically significant harm in terms of the interactional roles an AI counselor adopts, including perpetrator, instigator, facilitator, or enabler, combined with clinically grounded harm categories. Then, we propose MHSafeEval, a closed-loop, agent-based evaluation framework that formulates safety assessment as trajectory-level discovery of harm through adversarial multi-turn interactions, guided by role-aware modeling. Using R-MHSafe and MHSafeEval, we conduct a large-scale evaluation across state-of-the-art LLMs. Our results reveal substantial role-dependent and cumulative safety failures that are systematically missed by existing static benchmarks, and show that our framework significantly improves failure-mode coverage and diagnostic granularity.