# Training Therapeutic Judges and Multi-Agent Systems for Human-Aligned Mental Health Support

arXiv: https://arxiv.org/abs/2606.30887 · HTML: https://mlsanity.com/paper/2606.30887/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 20 of 20)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [JudgeBench: A Benchmark for Evaluating LLM-based Judges](https://mlsanity.com/paper/2410.12784/) (2024)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data](https://mlsanity.com/paper/2307.14385/) (2023)
- [The opportunities and risks of large language models in mental health](https://mlsanity.com/paper/2403.14814/) (2024)
- [MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance](https://mlsanity.com/paper/2503.13509/) (2025)
- [Kardia-R1: Unleashing LLMs to Reason toward Understanding and Empathy for Emotional Support via Rubric-as-Judge Reinforcement Learning](https://mlsanity.com/paper/2512.01282/) (2025)
- [When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation](https://mlsanity.com/paper/2510.19032/) (2025)
- [Rethinking Large Language Models in Mental Health Applications](https://mlsanity.com/paper/2311.11267/) (2023)
- [The Generative AI Paradox on Evaluation: What It Can Solve, It May Not Evaluate](https://mlsanity.com/paper/2402.06204/) (2024)
- [Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practice](https://mlsanity.com/paper/2501.00982/) (2025)
- [Assessing the Quality of Mental Health Support in LLM Responses through Multi-Attribute Human Evaluation](https://mlsanity.com/paper/2601.18630/) (2026)
- [Aligning Text, Code, and Vision: A Multi-Objective Reinforcement Learning Framework for Text-to-Visualization](https://mlsanity.com/paper/2601.04582/) (2026)
- [Reconciling Privacy and Explainability in High-Stakes: A Systematic Inquiry](https://mlsanity.com/paper/2412.20798/) (2024)
- [TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots](https://mlsanity.com/paper/2603.18008/) (2026)
- [Keyphrase Generative Representation of Youth Crisis Conversations Beyond Static Taxonomies](https://mlsanity.com/paper/2605.27546/) (2026)

## Abstract (arXiv)

Large language models show promise for mental health support, yet therapeutic quality improves only when evaluation functions as an actionable control signal rather than a passive metric. We introduce a framework that formulates therapeutic response generation as a decision-refinement problem driven by multi-dimensional, human-aligned evaluation. In Stage I, we introduce TheraJudge, an open-source therapeutic evaluator trained via preference-based optimization on human-annotated data to produce reliable judgments across 7 psychological dimensions. In Stage II, we introduce TheraAgent, which operationalizes TheraJudge's evaluations through a coordinated refinement process with specialized Critic, Coach, and Therapist roles that translate evaluative signals into targeted response revisions. Empirically, TheraJudge achieves strong agreement with clinician ratings, with intraclass correlation coefficients (ICC = 0.87-0.95), surpassing supervised baselines and strong closed-source judges, particularly on critical dimensions such as Safety, Relevance, and Empathy. Acting on these evaluations, TheraAgent yields a +0.43 improvement in human-rated therapeutic quality (on a 5-point scale) under blind evaluation, with 96% clinician inter-rater reliability. Low-quality responses (≤ 3) improve by +2.45 points with a 94% recovery rate, demonstrating targeted correction of unsafe outputs. Overall, our results indicate that effective alignment of mental-health LLMs stems from acting on human-aligned evaluation, rather than relying solely on stronger generation. We release code at https://github.com/vis-nlp/TheraAlign.