# Assessing the Quality of Mental Health Support in LLM Responses through Multi-Attribute Human Evaluation

arXiv: https://arxiv.org/abs/2601.18630 · HTML: https://mlsanity.com/paper/2601.18630/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 23 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Builds on (most cited first, 4 of 4)

- [MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance](https://mlsanity.com/paper/2503.13509/) (2025)
- [A Novel Nuanced Conversation Evaluation Framework for Large Language Models in Mental Health](https://mlsanity.com/paper/2403.09705/) (2024)
- [Interpretable Depression Detection from Social Media Text Using LLM-Derived Embeddings](https://mlsanity.com/paper/2506.06616/) (2025)
- [FAIIR: Building Toward A Conversational AI Agent Assistant for Youth Mental Health Service Provision](https://mlsanity.com/paper/2405.18553/) (2024)

## Built on by (most cited first, 6 of 6)

- [Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses](https://mlsanity.com/paper/2604.00014/) (2026)
- [Does Capability Transfer to Subjective Behavior -- and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation Paradigm](https://mlsanity.com/paper/2605.27914/) (2026)
- [Towards Understanding and Measuring COGNITIVE ATROPHY in LLM Behaviour](https://mlsanity.com/paper/2606.18129/) (2026)
- [One Year Later...The Harms Persist, But So Do We!](https://mlsanity.com/paper/2606.23884/) (2026)
- [Training Therapeutic Judges and Multi-Agent Systems for Human-Aligned Mental Health Support](https://mlsanity.com/paper/2606.30887/) (2026)
- [EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks](https://mlsanity.com/paper/2608.04549/) (2026)

## Abstract (arXiv)

The escalating global mental health crisis, marked by persistent treatment gaps, availability, and a shortage of qualified therapists, positions Large Language Models (LLMs) as a promising avenue for scalable support. While LLMs offer potential for accessible emotional assistance, their reliability, therapeutic relevance, and alignment with human standards remain challenging to address. This paper introduces a human-grounded evaluation methodology designed to assess LLM generated responses in therapeutic dialogue. Our approach involved curating a dataset of 500 mental health conversations from datasets with real-world scenario questions and evaluating the responses generated by nine diverse LLMs, including closed source and open source models. More specifically, these responses were evaluated by two psychiatric trained experts, who independently rated each on a 5 point Likert scale across a comprehensive 6 attribute rubric. This rubric captures Cognitive Support and Affective Resonance, providing a multidimensional perspective on therapeutic quality. Our analysis reveals that LLMs provide strong cognitive reliability by producing safe, coherent, and clinically appropriate information, but they demonstrate unstable affective alignment. Although closed source models (e.g., GPT-4o) offer balanced therapeutic responses, open source models show greater variability and emotional flatness. We reveal a persistent cognitive-affective gap and highlight the need for failure aware, clinically grounded evaluation frameworks that prioritize relational sensitivity alongside informational accuracy in mental health oriented LLMs. We advocate for balanced evaluation protocols with human in the loop that center on therapeutic sensitivity and provide a framework to guide the responsible design and clinical oversight of mental health oriented conversational AI.