# From "Help" to Helpful: A Hierarchical Assessment of LLMs in Mental e-Health Applications

arXiv: https://arxiv.org/abs/2602.18443 · HTML: https://mlsanity.com/paper/2602.18443/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 40 references that are in the map.
- First posted on arXiv 12 Jan 2026.

## Builds on (most cited first, 16 of 16)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [Human-like Summarization Evaluation with ChatGPT](https://mlsanity.com/paper/2304.02554/) (2023)
- [LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models](https://mlsanity.com/paper/2307.07889/) (2023)
- [A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review](https://mlsanity.com/paper/2405.02559/) (2024)
- [A Computational Framework for Behavioral Assessment of LLM Therapists](https://mlsanity.com/paper/2401.00820/) (2024)
- [This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation](https://mlsanity.com/paper/1906.03497/) (2019)
- [A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models](https://mlsanity.com/paper/2406.11289/) (2024)
- [ChatCounselor: A Large Language Models for Mental Health Support](https://mlsanity.com/paper/2309.15461/) (2023)
- [Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratings](https://mlsanity.com/paper/2308.02575/) (2023)
- [Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency](https://mlsanity.com/paper/2407.09519/) (2024)
- [Enhancing Psychological Counseling with Large Language Model: A Multifaceted Decision-Support System for Non-Professionals](https://mlsanity.com/paper/2308.15192/) (2023)
- [Are Large Language Models Reliable Argument Quality Annotators?](https://mlsanity.com/paper/2404.09696/) (2024)
- [Helping the Helper: Supporting Peer Counselors via AI-Empowered Practice and Feedback](https://mlsanity.com/paper/2305.08982/) (2023)
- [A Novel LLM-based Two-stage Summarization Approach for Long Dialogues](https://mlsanity.com/paper/2410.06520/) (2024)

## Built on by (most cited first, 1 of 1)

- [N"urnberg NLP at PsyDefDetect: Multi-Axis Voter Ensembles for Psychological Defence Mechanism Classification](https://mlsanity.com/paper/2605.07606/) (2026)

## Abstract (arXiv)

Psychosocial online counselling frequently encounters generic subject lines that impede efficient case prioritisation. This study evaluates eleven large language models generating six-word subject lines for German counselling emails through hierarchical assessment - first categorising outputs, then ranking within categories to enable manageable evaluation. Nine assessors (counselling professionals and AI systems) enable analysis via Krippendorff's α, Spearman's ρ, Pearson's r and Kendall's τ. Results reveal performance trade-offs between proprietary services and privacy-preserving open-source alternatives, with German fine-tuning consistently improving performance. The study addresses critical ethical considerations for mental health AI deployment including privacy, bias and accountability.