AI Safety Training Can be Clinically Harmful
Not on the map. Why?
Abstract (arXiv)
Large language models are being deployed as mental health support agents at scale, yet only 16% of LLM-based chatbot interventions have undergone rigorous clinical efficacy testing, and simulations reveal psychological deterioration in over one-third of cases. We evaluate four generative models on 250 Prolonged Exposure (PE) therapy scenarios and 146 CBT cognitive restructuring exercises (plus 29 severity-escalated variants), scored by a three-judge LLM panel. All models scored near-perfectly on surface acknowledgment (~0.91-1.00) while therapeutic appropriateness collapsed to 0.22-0.33 at the highest severity for three of four models, with protocol fidelity reaching zero for two. Under CBT severity escalation, one model's task completeness dropped from 92% to 71% while the frontier model's safety-interference score fell from 0.99 to 0.61. We identify a systematic, modality-spanning failure: RLHF safety alignment disrupts the therapeutic mechanism of action by grounding patients during imaginal exposure, offering false reassurance, inserting crisis resources into controlled exercises, and refusing to challenge distorted cognitions mentioning self-harm in PE; and through task abandonment or safety-preamble insertion during CBT cognitive restructuring. These findings motivate a five-axis evaluation framework (protocol fidelity, hallucination risk, behavioral consistency, crisis safety, demographic robustness), mapped onto FDA SaMD and EU AI Act requirements. We argue that no AI mental health system should proceed to deployment without passing multi-axis evaluation across all five dimensions.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 19 of its 66 references that are in the map. First posted on arXiv 25 Apr 2026.
Builds on
19 of 66 references.
References · 19
- 2024Polaris: A Safety-focused LLM Constellation Architecture for Healthcarecited by 28
- 2025A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domainscited by 28
- 2024MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applicationscited by 25
- 2024LLM-based Conversational AI Therapist for Daily Functioning Screening and Psychotherapeutic Intervention via Everyday Smart Devicescited by 21
- 2025EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safetycited by 21
- 2025Thousand Voices of Trauma: A Large-Scale Synthetic Dataset for Modeling Prolonged Exposure Therapy Conversationscited by 11
- 2022Privacy Sensitive Speech Analysis Using Federated Learning to Assess Depressioncited by 8
- 2025SAGE: A Generic Framework for LLM Safety Evaluationcited by 8
- 2025How Real Are Synthetic Therapy Conversations? Evaluating Fidelity in Prolonged Exposure Dialoguescited by 8
- 2025Attributional Safety Failures in Large Language Models under Code-Mixed Perturbationscited by 6
- 2025The Pursuit of Empathy: Evaluating Small Language Models for PTSD Dialogue Supportcited by 6
- 2025From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysiscited by 4
7 more
- 2025Domain-Specific Constitutional AI: Enhancing Safety in LLM-Powered Mental Health Chatbotscited by 4
- 2026AI Chatbot Suicide Risk Detection and Response: Human Validation Study of the Open-Source VERA-MH Safety Evaluationcited by 4
- 2024The Efficacy of Conversational Artificial Intelligence in Rectifying the Theory of Mind and Autonomy Biases: Comparative Analysiscited by 3
- 2025Fine-Tuning Large Audio-Language Models with LoRA for Precise Temporal Localization of Prolonged Exposure Therapy Elementscited by 3
- 2025Fact-Controlled Diagnosis of Hallucinations in Medical Text Summarizationcited by 2
- 2022Automated Fidelity Assessment for Strategy Training in Inpatient Rehabilitation using Natural Language Processingcited by 1
- 2026JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Modelscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026ComBodied Agents: a New Paradigm of Human-Centric Agentic AI11 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2026Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming
- 2026TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots
- 2026Beyond Simulations: What 20,000 Real Conversations Reveal About Mental Health AI Safety
- 2026When Vocabulary Comprehension Fails Clinical Reasoning: Evaluating Therapy Bots' Safety Risks for Generation Alpha
- 2026Alignment Plausibility: A New Standard for Assuring AI in Healthcare
- 2026Retrieval-Augmented Generation in LLMs for Mental Health: Quantifying the Incremental Contribution of Retrieval Within a Layered Safety Architecture
- 2025MindEval: Benchmarking Language Models on Multi-turn Mental Health Support
- 2026MindGuard: Guardrail Classifiers for Multi-Turn Mental Health Support