mlsanityData to 7 Oct 2026

Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation

arXiv 2406.11852 ↗First posted 2 Apr 2024cs.CY cs.AI cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

Amidst the growing interest in developing task-autonomous AI for automated mental health care, this paper addresses the ethical and practical challenges associated with the issue and proposes a structured framework that delineates levels of autonomy, outlines ethical requirements, and defines beneficial default behaviors for AI agents in the context of mental health support. We also evaluate fourteen state-of-the-art language models (ten off-the-shelf, four fine-tuned) using 16 mental health-related questionnaires designed to reflect various mental health conditions, such as psychosis, mania, depression, suicidal thoughts, and homicidal tendencies. The questionnaire design and response evaluations were conducted by mental health clinicians (M.D.s). We find that existing language models are insufficient to match the standard provided by human professionals who can navigate nuances and appreciate context. This is due to a range of issues, including overly cautious or sycophantic responses and the absence of necessary safeguards. Alarmingly, we find that most of the tested models could cause harm if accessed in mental health emergencies, failing to protect users and potentially exacerbating existing symptoms. We explore solutions to enhance the safety of current models. Before the release of increasingly task-autonomous AI systems in mental health, it is crucial to ensure that these models can reliably detect and manage symptoms of common psychiatric disorders to prevent harm to users. This involves aligning with the ethical framework and default behaviors outlined in our study. We contend that model developers are responsible for refining their systems per these guidelines to safeguard against the risks posed by current AI technologies to user mental health and safety. Trigger warning: Contains and discusses examples of sensitive mental health topics, including suicide and self-harm.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 48 of its 107 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

48 of 107 references.

References · 48

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  9. 2023Mistral 7Bcited by 3,065
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
36 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  5. 2022Red Teaming Language Models with Language Modelscited by 983
  6. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  8. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  9. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  10. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  11. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  12. 2023Are aligned neural networks adversarially aligned?cited by 305
  13. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  14. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  15. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  16. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  17. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  18. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  19. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  20. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  21. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  22. 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 92
  23. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 87
  24. 2023DICES Dataset: Diversity in Conversational AI Evaluation for Safetycited by 86
  25. 2020Trojaning Language Models for Fun and Profitcited by 80
  26. 2024Towards Conversational Diagnostic AIcited by 79
  27. 2023Efficient Benchmarking of Language Modelscited by 71
  28. 2022Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Supportcited by 68
  29. 2023MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Modelscited by 63
  30. 2024Escalation Risks from Language Models in Military and Diplomatic Decision-Makingcited by 61
  31. 2024A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Modelscited by 45
  32. 2023Analyzing And Editing Inner Mechanisms Of Backdoored Language Modelscited by 22
  33. 2024Human vs. Machine: Behavioral Differences Between Expert Humans and Language Models in Wargame Simulationscited by 20
  34. 2023Towards Healthy AI: Large Language Models Need Therapists Toocited by 13
  35. 2023ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Modelscited by 12
  36. 2023Human Control: Definitions and Algorithmscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 10; 2026: 10.

27 papers.

Sort:

By area:

  1. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  2. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
  3. 2025Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providerscited by 67
  4. 2024Aloe: A Family of Fine-tuned Open Healthcare LLMscited by 20
  5. 2025CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answeringcited by 20
  6. 2024More than Marketing? On the Information Value of AI Benchmarks for Practitionerscited by 16
  7. 2024Acceptable Use Policies for Foundation Modelscited by 12
  8. 2024CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapycited by 12
  9. 2024Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulationscited by 11
  10. 2025CRADLE Bench: A Clinician-Annotated Benchmark for Multi-Faceted Mental Health Crisis and Safety Risk Detectioncited by 8
  11. 2025Just a Scratch: Enhancing LLM Capabilities for Self-harm Detection through Intent Differentiation and Emoji Interpretationcited by 7
  12. 2024Markovian Transformers for Informative Language Modelingcited by 4
15 more
  1. 2026Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counselingcited by 4
  2. 2025DICE: A Framework for Dimensional and Contextual Evaluation of Language Modelscited by 3
  3. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Modelscited by 3
  4. 2026Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosiscited by 3
  5. 2025SAIF: A Comprehensive Framework for Evaluating the Risks of Generative AI in the Public Sectorcited by 2
  6. 2025Seeking Late Night Life Lines: Experiences of Conversational AI Use in Mental Health Crisiscited by 2
  7. 2026Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testingcited by 2
  8. 2026Understanding Risk and Dependency in AI Chatbot Use from User Discoursecited by 2
  9. 2026SEED-SET: Scalable Evolving Experimental Design for System-level Ethical Testingcited by 1
  10. 2026Receptiveness, Not Sycophancy: Distinguishing Engagement from Deference in Language Modelscited by 1
  11. 2025When LLM Therapists Become Salespeople: Evaluating Large Language Models for Ethical Motivational Interviewingcited by 0
  12. 2025Two-Stage Voting for Robust and Efficient Suicide Risk Detection on Social Mediacited by 0
  13. 2026Multi-Objective Alignment of Language Models for Personalized Psychotherapycited by 0
  14. 2026Psychological Influences of Conversational AI: Research and Design Directions for Reducing Harm and Promoting Well-Beingcited by 0
  15. 2026Right Words, Wrong Moment: A Clinician-Grounded Analysis of Distress in 19,930 Conversations between Young People and ChatGPTcited by 0