# PsychEthicsBench: Evaluating Large Language Models Against Australian Mental Health Ethics

arXiv: https://arxiv.org/abs/2601.03578 · HTML: https://mlsanity.com/paper/2601.03578/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 21 of its 53 references that are in the map.
- First posted on arXiv 7 Jan 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Medical LLM alignment](https://mlsanity.com/alignment/medical-llm-alignment/)

## Builds on (most cited first, 21 of 21)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs](https://mlsanity.com/paper/2410.05295/) (2024)
- [Cognitive Bias in Decision-Making with LLMs](https://mlsanity.com/paper/2403.00811/) (2024)
- [Med42-v2: A Suite of Clinical LLMs](https://mlsanity.com/paper/2408.06142/) (2024)
- [Measuring Political Bias in Large Language Models: What Is Said and How It Is Said](https://mlsanity.com/paper/2403.18932/) (2024)
- [Building Emotional Support Chatbots in the Era of LLMs](https://mlsanity.com/paper/2308.11584/) (2023)
- [EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysis](https://mlsanity.com/paper/2401.08508/) (2024)
- [Cactus: Towards Psychological Counseling Conversations using Cognitive Behavioral Theory](https://mlsanity.com/paper/2407.03103/) (2024)
- [Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversation](https://mlsanity.com/paper/2402.13211/) (2024)
- [ESCoT: Towards Interpretable Emotional Support Dialogue Systems](https://mlsanity.com/paper/2406.10960/) (2024)
- [PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation](https://mlsanity.com/paper/2407.05721/) (2024)
- [SweetieChat: A Strategy-Enhanced Role-playing Framework for Diverse Scenarios Handling Emotional Support Agent](https://mlsanity.com/paper/2412.08389/) (2024)
- [EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety](https://mlsanity.com/paper/2504.09689/) (2025)
- [Crisp: Cognitive Restructuring of Negative Thoughts through Multi-turn Supportive Dialogues](https://mlsanity.com/paper/2504.17238/) (2025)
- [PsyGUARD: An Automated System for Suicide Detection and Risk Assessment in Psychological Counseling](https://mlsanity.com/paper/2409.20243/) (2024)
- [MedEthicsQA: A Comprehensive Question Answering Benchmark for Medical Ethics Evaluation of LLMs](https://mlsanity.com/paper/2506.22808/) (2025)
- [MedFact: A Large-scale Chinese Dataset for Evidence-based Medical Fact-checking of LLM Responses](https://mlsanity.com/paper/2509.17436/) (2025)
- [Emotional Support with LLM-based Empathetic Dialogue Generation](https://mlsanity.com/paper/2507.12820/) (2025)
- [Can LLMs Address Mental Health Questions? A Comparison with Human Therapists](https://mlsanity.com/paper/2509.12102/) (2025)
- [Roleplaying with Structure: Synthetic Therapist-Client Conversation Generation from Questionnaires](https://mlsanity.com/paper/2510.25384/) (2025)

## Built on by (most cited first, 4 of 4)

- [Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling](https://mlsanity.com/paper/2604.04842/) (2026)
- [SDARE-Bench: Evaluating Large Language Models on Conversational Stigma Detection and Response in Dyadic and Group Dialogue](https://mlsanity.com/paper/2609.01548/) (2026)
- [SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment](https://mlsanity.com/paper/2609.20584/) (2026)
- [DistressBench: Evaluating Multi-Dimensional Crisis Support in Large Language Models](https://mlsanity.com/paper/2610.12758/) (2026)

## Abstract (arXiv)

The increasing integration of large language models (LLMs) into mental health applications necessitates robust frameworks for evaluating professional safety alignment. Current evaluative approaches primarily rely on refusal-based safety signals, which offer limited insight into the nuanced behaviors required in clinical practice. In mental health, clinically inadequate refusals can be perceived as unempathetic and discourage help-seeking. To address this gap, we move beyond refusal-centric metrics and introduce textttPsychEthicsBench, the first principle-grounded benchmark based on Australian psychology and psychiatry guidelines, designed to evaluate LLMs' ethical knowledge and behavioral responses through multiple-choice and open-ended tasks with fine-grained ethicality annotations. Empirical results across 14 models reveal that refusal rates are poor indicators of ethical behavior, revealing a significant divergence between safety triggers and clinical appropriateness. Notably, we find that domain-specific fine-tuning can degrade ethical robustness, as several specialized models underperform their base backbones in ethical alignment. PsychEthicsBench provides a foundation for systematic, jurisdiction-aware evaluation of LLMs in mental health, encouraging more responsible development in this domain.