# Exploring Safety Alignment Evaluation of LLMs in Chinese Mental Health Dialogues via LLM-as-Judge

arXiv: https://arxiv.org/abs/2508.08236 · HTML: https://mlsanity.com/paper/2508.08236/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 11 Aug 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Medical LLM alignment](https://mlsanity.com/alignment/medical-llm-alignment/)

## Builds on (most cited first, 19 of 19)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://mlsanity.com/paper/2406.12624/) (2024)
- [SoulChat: Improving LLMs' Empathy, Listening, and Comfort Abilities through Fine-tuning with Multi-turn Empathy Conversations](https://mlsanity.com/paper/2311.00273/) (2023)
- [PsyQA: A Chinese Dataset for Generating Long Counseling Text for Mental Health Support](https://mlsanity.com/paper/2106.01702/) (2021)
- [CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling](https://mlsanity.com/paper/2405.16433/) (2024)
- [Out of the BLEU: how should we assess quality of the Code Generation models?](https://mlsanity.com/paper/2208.03133/) (2022)
- [HealMe: Harnessing Cognitive Reframing in Large Language Models for Psychotherapy](https://mlsanity.com/paper/2403.05574/) (2024)
- [EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety](https://mlsanity.com/paper/2504.09689/) (2025)
- [PsyEval: A Suite of Mental Health Related Tasks for Evaluating Large Language Models](https://mlsanity.com/paper/2311.09189/) (2023)
- [Supervised Learning and Large Language Model Benchmarks on Mental Health Datasets: Cognitive Distortions and Suicidal Risks in Chinese Social Media](https://mlsanity.com/paper/2309.03564/) (2023)
- [MoodAngels: A Retrieval-augmented Multi-agent Framework for Psychiatry Diagnosis](https://mlsanity.com/paper/2506.03750/) (2025)
- [A Framework for Evaluating Appropriateness, Trustworthiness, and Safety in Mental Wellness AI Chatbots](https://mlsanity.com/paper/2407.11387/) (2024)
- [Unlocking LLMs: Addressing Scarce Data and Bias Challenges in Mental Health](https://mlsanity.com/paper/2412.12981/) (2024)
- [PsychBench: A comprehensive and professional benchmark for evaluating the performance of LLM-assisted psychiatric clinical practice](https://mlsanity.com/paper/2503.01903/) (2025)
- [Evaluating Large Language Models in Crisis Detection: A Real-World Benchmark from Psychological Support Hotlines](https://mlsanity.com/paper/2506.01329/) (2025)

## Built on by (most cited first, 4 of 4)

- [The Supportiveness-Safety Tradeoff in LLM Well-Being Agents](https://mlsanity.com/paper/2602.04487/) (2026)
- [Towards Understanding and Measuring COGNITIVE ATROPHY in LLM Behaviour](https://mlsanity.com/paper/2606.18129/) (2026)
- [Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy](https://mlsanity.com/paper/2608.21325/) (2026)
- [From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Health](https://mlsanity.com/paper/2609.25186/) (2026)

## Abstract (arXiv)

Evaluating the safety alignment of LLM responses in high-risk mental health dialogues is particularly difficult due to missing gold-standard answers and the ethically sensitive nature of these interactions. To address this challenge, we propose PsyCrisis-Bench, a reference-free evaluation benchmark based on real-world Chinese mental health dialogues. It evaluates whether the model responses align with the safety principles defined by experts. Specifically designed for settings without standard references, our method adopts a prompt-based LLM-as-Judge approach that conducts in-context evaluation using expert-defined reasoning chains grounded in psychological intervention principles. We employ binary point-wise scoring across multiple safety dimensions to enhance the explainability and traceability of the evaluation. Additionally, we present a manually curated, high-quality Chinese-language dataset covering self-harm, suicidal ideation, and existential distress, derived from real-world online discourse. Experiments on 3600 judgments show that our method achieves the highest agreement with expert assessments and produces more interpretable evaluation rationales compared to existing approaches. Our dataset and evaluation tool are publicly available to facilitate further research.