mlsanityData to 7 Oct 2026

A Computational Framework for Behavioral Assessment of LLM Therapists

arXiv 2401.00820 ↗First posted 1 Jan 2024cs.CL cs.HCcited by 61

Not on the map. Why?

Abstract (arXiv)

The emergence of large language models (LLMs) like ChatGPT has increased interest in their use as therapists to address mental health challenges and the widespread lack of access to care. However, experts have emphasized the critical need for systematic evaluation of LLM-based mental health interventions to accurately assess their capabilities and limitations. Here, we propose BOLT, a proof-of-concept computational framework to systematically assess the conversational behavior of LLM therapists. We quantitatively measure LLM behavior across 13 psychotherapeutic approaches with in-context learning methods. Then, we compare the behavior of LLMs against high- and low-quality human therapy. Our analysis based on Motivational Interviewing therapy reveals that LLMs often resemble behaviors more commonly exhibited in low-quality therapy rather than high-quality therapy, such as offering a higher degree of problem-solving advice when clients share emotions. However, unlike low-quality therapy, LLMs reflect significantly more upon clients' needs and strengths. Our findings caution that LLM therapists still require further research for consistent, high-quality care.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 19 of its 72 references that are in the map. First posted on arXiv 1 Jan 2024.

Builds on

19 of 72 references.

References · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  7. 2021Ethical and social risks of harm from Language Modelscited by 957
  8. 2023Whose Opinions Do Language Models Reflect?cited by 759
  9. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 143
  10. 2023Cognitive Reframing of Negative Thoughts through Human-Language Model Interactioncited by 77
  11. 2022Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Supportcited by 68
  12. 2023Facilitating Self-Guided Mental Health Interventions Through Human-Language Model Interaction: A Case Study of Cognitive Restructuringcited by 53
7 more
  1. 2023Grounding Gaps in Language Model Generationscited by 45
  2. 2023Benefits and Harms of Large Language Models in Digital Mental Healthcited by 39
  3. 2021Speaker and Time-aware Joint Contextual Learning for Dialogue-act Classification in Counselling Conversationscited by 31
  4. 2023Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare deliverycited by 30
  5. 2019Observing Dialogue in Therapy: Categorizing and Forecasting Behavioral Codescited by 25
  6. 2022Gendered Mental Health Stigma in Masked Language Modelscited by 17
  7. 2023VERVE: Template-based ReflectiVE Rewriting for MotiVational IntErviewingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 25; 2026: 15.

61 papers.

Sort:

By area:

  1. 2024MindScape Study: Integrating LLM and Behavioral Sensing for Personalized AI-Driven Journaling Experiencescited by 24
  2. 2024The Typing Cure: Experiences with Large Language Model Chatbots for Mental Health Supportcited by 23
  3. 2024Towards a Client-Centered Assessment of LLM Therapists by Client Simulationcited by 23
  4. 2024Can AI Relate: Testing Large Language Model Response for Mental Health Supportcited by 21
  5. 2025When Large Language Models are Reliable for Judging Empathic Communicationcited by 21
  6. 2025CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answeringcited by 20
  7. 2024Understanding the Therapeutic Relationship between Counselors and Clients in Online Text-based Counseling using LLMscited by 19
  8. 2024PATIENT-Psi: Using Large Language Models to Simulate Patients for Training Mental Health Professionalscited by 19
  9. 2024Do Large Language Models Align with Core Mental Health Counseling Competencies?cited by 19
  10. 2024Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselorscited by 16
  11. 2024Contextual AI Journaling: Integrating LLM and Time Series Behavioral Sensing Technology to Promote Self-Reflection and Well-being using the MindScape Appcited by 14
  12. 2025Consistent Client Simulation for Motivational Interviewing-based Counselingcited by 14
38 more
  1. 2024Social Skill Training with Large Language Modelscited by 13
  2. 2024AdvisorQA: Towards Helpful and Harmless Advice-seeking Question Answering with Collective Intelligencecited by 12
  3. 2025A Survey of Large Language Models in Psychotherapy: Current Landscape and Future Directionscited by 12
  4. 2025CAMI: A Counselor Agent Supporting Motivational Interviewing through State Inference and Topic Explorationcited by 11
  5. 2024Towards Collaborative Intelligence: Propagating Intentions and Reasoning for Multi-Agent Coordination with Large Language Modelscited by 10
  6. 2025AutoCBT: An Autonomous Multi-agent Framework for Cognitive Behavioral Therapy in Psychological Counselingcited by 10
  7. 2025AI with Emotions: Exploring Emotional Expressions in Large Language Modelscited by 10
  8. 2024Digital Diagnostics: The Potential Of Large Language Models In Recognizing Symptoms Of Common Illnessescited by 8
  9. 2024MentalAgora: A Gateway to Advanced Personalized Care in Mental Health through Multi-Agent Debating and Attribute Controlcited by 8
  10. 2025How Real Are Synthetic Therapy Conversations? Evaluating Fidelity in Prolonged Exposure Dialoguescited by 8
  11. 2025Multimodal Cognitive Reframing Therapy via Multi-hop Psychotherapeutic Reasoningcited by 7
  12. 2025Can You Share Your Story? Modeling Clients' Metacognition and Openness for LLM Therapist Evaluationcited by 7
  13. 2026Computational Hermeneutics: Evaluating generative AI as a cultural technologycited by 7
  14. 2025MAGneT: Coordinated Multi-Agent Generation of Synthetic Multi-Turn Mental Health Counseling Sessionscited by 6
  15. 2024ExploreSelf: Fostering User-driven Exploration and Reflection on Personal Challenges with Adaptive Guidance by Large Language Modelscited by 5
  16. 2024Multi-Session Client-Centered Treatment Outcome Evaluation in Psychotherapycited by 5
  17. 2025Behavioral Fingerprinting of Large Language Modelscited by 5
  18. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  19. 2025From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysiscited by 4
  20. 2025Examining Spanish Counseling with MIDAS: a Motivational Interviewing Dataset in Spanishcited by 4
  21. 2025DiaCBT: A Long-Periodic Dialogue Corpus Guided by Cognitive Conceptualization Diagram for CBT-based Psychological Counselingcited by 4
  22. 2026PatientHub: A Unified Framework for Patient Simulationcited by 4
  23. 2024Enhancing Depression-Diagnosis-Oriented Chat with Psychological State Trackingcited by 2
  24. 2024Explaining GPTs' Schema of Depression: A Machine Behavior Analysiscited by 2
  25. 2025Leveraging LLMs for Mental Health: Detection and Recommendations from Social Discussionscited by 2
  26. 2024EMMI -- Empathic Multimodal Motivational Interviews Dataset: Analyses and Annotationscited by 1
  27. 2024Prompt and Prejudicecited by 1
  28. 2025Wearable Meets LLM for Stress Management: A Duoethnographic Study Integrating Wearable-Triggered Stressors and LLM Chatbots for Personalized Interventionscited by 1
  29. 2025Can Large Language Models Integrate Spatial Data? Empirical Insights into Reasoning Strengths and Computational Weaknessescited by 1
  30. 2026coTherapist: A Behavior-Aligned Small Language Model to Support Mental Healthcare Expertscited by 1
  31. 2025AI for Proactive Mental Health: A Multi-Institutional, Longitudinal, Randomized Controlled Trialcited by 1
  32. 2026A Checklist for Trustworthy, Safe, and User-Friendly Mental Health Chatbotscited by 1
  33. 2026From "Help" to Helpful: A Hierarchical Assessment of LLMs in Mental e-Health Applicationscited by 1
  34. 2024Therapy as an NLP Task: Comparing LLMs and Human Peers Behaviors in CBT Sessionscited by 0
  35. 2025When and How to Integrate Multimodal Large Language Models in College Psychotherapy: Perspectives from Multi-stakeholderscited by 0
  36. 2025CALLM: Understanding Cancer Survivors' Emotions and Intervention Opportunities via Mobile Diaries and Context-Aware Language Modelscited by 0
  37. 2025When LLM Therapists Become Salespeople: Evaluating Large Language Models for Ethical Motivational Interviewingcited by 0
  38. 2025Visually grounded emotion regulation via diffusion models and user-driven reappraisalcited by 0

50 most cited of 61 All 61 (JSON)