Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming
On the map: LLM safety and security → Safety alignment, refusal and guardrails
Abstract (arXiv)
Large Language Models (LLMs) are increasingly utilized for mental health support; however, current safety benchmarks often fail to detect the complex, longitudinal risks inherent in therapeutic dialogue. We introduce an evaluation framework that pairs AI psychotherapists with simulated patient agents equipped with dynamic cognitive-affective models and assesses therapy session simulations against a comprehensive quality of care and risk ontology. We apply this framework to a high-impact test case, Alcohol Use Disorder, evaluating six AI agents (including ChatGPT, Gemini, and Character AI) against a clinically-validated cohort of 15 patient personas representing diverse clinical phenotypes. Our large-scale simulation (N=369 sessions) reveals critical safety gaps in the use of AI for mental health support. We identify specific iatrogenic risks, including the validation of patient delusions ("AI Psychosis") and failure to de-escalate suicide risk. Finally, we validate an interactive data visualization dashboard with diverse stakeholders, including AI engineers and red teamers, mental health professionals, and policy experts (N=9), demonstrating that this framework effectively enables stakeholders to audit the "black box" of AI psychotherapy. These findings underscore the critical safety risks of AI-provided mental health support and the necessity of simulation-based clinical red teaming before deployment.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 44 of its 215 references that are in the map. First posted on arXiv 23 Feb 2026.
Builds on
44 of 215 references.
References · 44
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
- 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
- 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
- 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
- 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
- 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
- 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
- 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 134
- 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
- 2025SycEval: Evaluating LLM Sycophancycited by 101
32 more
- 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
- 2024Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principlescited by 72
- 2025Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providerscited by 67
- 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Taskscited by 60
- 2023Rehearsal: Simulating Conflict to Teach Conflict Resolutioncited by 54
- 2024Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challengescited by 48
- 2024Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulatorcited by 36
- 2024Large Language Models Produce Responses Perceived to be Empathiccited by 36
- 2023The Capability of Large Language Models to Measure Psychiatric Functioningcited by 25
- 2024From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agentscited by 25
- 2024Six Fallacies in Substituting Large Language Models for Human Participantscited by 24
- 2023Helping the Helper: Supporting Peer Counselors via AI-Empowered Practice and Feedbackcited by 23
- 2025Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMscited by 22
- 2025EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safetycited by 21
- 2024Lived Experience Not Found: LLMs Struggle to Align with Experts on Addressing Adverse Drug Reactions from Psychiatric Medication Usecited by 17
- 2025Scaffolding Empathy: Training Counselors with Simulated Patients and Utterance-level Performance Visualizationscited by 14
- 2024STAR: SocioTechnical Approach to Red Teaming Language Modelscited by 13
- 2025Psi-Arena: Interactive Assessment and Optimization of LLM-based Psychological Counselors with Tripartite Feedbackcited by 10
- 2024Large language models for mental healthcited by 9
- 2024A Novel Nuanced Conversation Evaluation Framework for Large Language Models in Mental Healthcited by 7
- 2025Just a Scratch: Enhancing LLM Capabilities for Self-harm Detection through Intent Differentiation and Emoji Interpretationcited by 7
- 2025TalkDep: Clinically Grounded LLM Personas for Conversation-Centric Depression Screeningcited by 6
- 2025A Risk Ontology for Evaluating AI-Powered Psychotherapy Virtual Agentscited by 5
- 2025AI Agents for Conversational Patient Triage: Preliminary Simulation-Based Evaluation with Real-World EHR Datacited by 5
- 2025DPRF: A Generalizable Dynamic Persona Refinement Framework for Optimizing Behavior Alignment Between Personalized LLM Role-Playing Agents and Humanscited by 5
- 2025Scaling Personality Control in LLMs with Big Five Scaler Promptscited by 4
- 2026Beyond Simulations: What 20,000 Real Conversations Reveal About Mental Health AI Safetycited by 4
- 2026PatientHub: A Unified Framework for Patient Simulationcited by 4
- 2025DialogGuard: Multi-Agent Psychosocial Safety Evaluation of Sensitive LLM Responsescited by 3
- 2024Mixed Chain-of-Psychotherapies for Emotional Support Chatbotcited by 2
- 2026PsychePass: Calibrating LLM Therapeutic Competence via Trajectory-Anchored Tournamentscited by 2
- 2025LLM Use for Mental Health: Crowdsourcing Users' Sentiment-based Perspectives and Values from Social Discussionscited by 1
Built on by
Citing papers per year: 2026: 3.
3 papers.