# Multi-Objective Alignment of Language Models for Personalized Psychotherapy

arXiv: https://arxiv.org/abs/2602.16053 · HTML: https://mlsanity.com/paper/2602.16053/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 8 of its 31 references that are in the map.
- First posted on arXiv 17 Feb 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Medical LLM alignment](https://mlsanity.com/alignment/medical-llm-alignment/)

## Builds on (most cited first, 8 of 8)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support](https://mlsanity.com/paper/2009.08441/) (2020)
- [Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization](https://mlsanity.com/paper/2310.03708/) (2023)
- [Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation](https://mlsanity.com/paper/2406.11852/) (2024)
- [ModelCitizens: Representing Community Voices in Online Safety](https://mlsanity.com/paper/2507.05455/) (2025)
- [Generative AI Purpose-built for Social and Mental Health: A Real-World Pilot](https://mlsanity.com/paper/2511.11689/) (2025)

## Abstract (arXiv)

Mental health disorders affect over 1 billion people worldwide, yet access to care remains limited by workforce shortages and cost constraints. While AI systems show therapeutic promise, current alignment approaches optimize objectives independently, failing to balance patient preferences with clinical safety. We survey 335 individuals with lived mental health experience to collect preference rankings across therapeutic dimensions, then develop a multi-objective alignment framework using direct preference optimization. We train reward models for six criteria -- empathy, safety, active listening, self-motivated change, trust/rapport, and patient autonomy -- and systematically compare multi-objective approaches against single-objective optimization, supervised fine-tuning, and parameter merging. Multi-objective DPO (MODPO) achieves superior balance (77.6% empathy, 62.6% safety) compared to single-objective optimization (93.6% empathy, 47.8% safety), and therapeutic criteria outperform general communication principles by 17.2%. Blinded clinician evaluation confirms MODPO is consistently preferred, with LLM-evaluator agreement comparable to inter-clinician reliability.