# Towards a Client-Centered Assessment of LLM Therapists by Client Simulation

arXiv: https://arxiv.org/abs/2406.12266 · HTML: https://mlsanity.com/paper/2406.12266/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 75 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Builds on (most cited first, 5 of 5)

- [PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits](https://mlsanity.com/paper/2305.02547/) (2023)
- [A Computational Framework for Behavioral Assessment of LLM Therapists](https://mlsanity.com/paper/2401.00820/) (2024)
- [Benefits and Harms of Large Language Models in Digital Mental Health](https://mlsanity.com/paper/2311.14693/) (2023)
- [Leveraging Large Language Model as Simulated Patients for Clinical Education](https://mlsanity.com/paper/2404.13066/) (2024)
- [Understanding the Therapeutic Relationship between Counselors and Clients in Online Text-based Counseling using LLMs](https://mlsanity.com/paper/2402.11958/) (2024)

## Built on by (most cited first, 24 of 24)

- [Interactive Agents: Simulating Counselor-Client Psychological Counseling via Role-Playing LLM-to-LLM Interactions](https://mlsanity.com/paper/2408.15787/) (2024)
- [PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactions](https://mlsanity.com/paper/2505.17818/) (2025)
- [CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering](https://mlsanity.com/paper/2506.08584/) (2025)
- [Towards Dynamic Theory of Mind: Evaluating LLM Adaptation to Temporal Evolution of Human States](https://mlsanity.com/paper/2505.17663/) (2025)
- [Consistent Client Simulation for Motivational Interviewing-based Counseling](https://mlsanity.com/paper/2502.02802/) (2025)
- [CAMI: A Counselor Agent Supporting Motivational Interviewing through State Inference and Topic Exploration](https://mlsanity.com/paper/2502.02807/) (2025)
- [Can You Share Your Story? Modeling Clients' Metacognition and Openness for LLM Therapist Evaluation](https://mlsanity.com/paper/2507.19643/) (2025)
- [Multi-Session Client-Centered Treatment Outcome Evaluation in Psychotherapy](https://mlsanity.com/paper/2410.05824/) (2024)
- [Analyzing Nobel Prize Literature with Large Language Models](https://mlsanity.com/paper/2410.18142/) (2024)
- [PatientHub: A Unified Framework for Patient Simulation](https://mlsanity.com/paper/2602.11684/) (2026)
- [Eeyore: Realistic Depression Simulation via Supervised and Preference Optimization](https://mlsanity.com/paper/2503.00018/) (2025)
- [Envisioning an AI-Enhanced Mental Health Ecosystem](https://mlsanity.com/paper/2503.14883/) (2025)
- [PsychePass: Calibrating LLM Therapeutic Competence via Trajectory-Anchored Tournaments](https://mlsanity.com/paper/2601.20330/) (2026)
- [Auditing Support Strategies in LLMs through Grounded Multi-Turn Social Simulation](https://mlsanity.com/paper/2604.17079/) (2026)
- [Enhancing User Engagement in Socially-Driven Dialogue through Interactive LLM Alignments](https://mlsanity.com/paper/2506.21497/) (2025)
- [PsyCLIENT: Client Simulation via Conversational Trajectory Modeling for Trainee Practice and Model Evaluation in Mental Health Counseling](https://mlsanity.com/paper/2601.07312/) (2026)
- [An LLM-based Simulation Framework for Embodied Conversational Agents in Psychological Counseling](https://mlsanity.com/paper/2410.22041/) (2024)
- [When LLM Therapists Become Salespeople: Evaluating Large Language Models for Ethical Motivational Interviewing](https://mlsanity.com/paper/2503.23566/) (2025)
- ["Is This Really a Human Peer Supporter?": Misalignments Between Peer Supporters and Experts in LLM-Supported Interactions](https://mlsanity.com/paper/2506.09354/) (2025)
- ["I Said Things I Needed to Hear Myself": Peer Support as an Emotional, Organisational, and Sociotechnical Practice in Singapore](https://mlsanity.com/paper/2506.09362/) (2025)
- [Tell Me: An LLM-powered Mental Well-being Assistant with RAG, Synthetic Dialogue Generation, and Agentic Planning](https://mlsanity.com/paper/2511.14445/) (2025)
- [Understanding Mental States to Guide Social Influence in Multi-Person Group Dialogue](https://mlsanity.com/paper/2601.13687/) (2026)
- [CARE: An Explainable Computational Framework for Assessing Client-Perceived Therapeutic Alliance Using Large Language Models](https://mlsanity.com/paper/2602.20648/) (2026)
- [Evaluation of Motivational Interviewing Counsellors with Task-Aware Multi-Stage LLM-Based Simulated Clients](https://mlsanity.com/paper/2608.07499/) (2026)

## Abstract (arXiv)

Although there is a growing belief that LLMs can be used as therapists, exploring LLMs' capabilities and inefficacy, particularly from the client's perspective, is limited. This work focuses on a client-centered assessment of LLM therapists with the involvement of simulated clients, a standard approach in clinical medical education. However, there are two challenges when applying the approach to assess LLM therapists at scale. Ethically, asking humans to frequently mimic clients and exposing them to potentially harmful LLM outputs can be risky and unsafe. Technically, it can be difficult to consistently compare the performances of different LLM therapists interacting with the same client. To this end, we adopt LLMs to simulate clients and propose ClientCAST, a client-centered approach to assessing LLM therapists by client simulation. Specifically, the simulated client is utilized to interact with LLM therapists and complete questionnaires related to the interaction. Based on the questionnaire results, we assess LLM therapists from three client-centered aspects: session outcome, therapeutic alliance, and self-reported feelings. We conduct experiments to examine the reliability of ClientCAST and use it to evaluate LLMs therapists implemented by Claude-3, GPT-3.5, LLaMA3-70B, and Mixtral 8*7B. Codes are released at https://github.com/wangjs9/ClientCAST.