# Therapy as an NLP Task: Comparing LLMs and Human Peers Behaviors in CBT Sessions

arXiv: https://arxiv.org/abs/2409.02244 · HTML: https://mlsanity.com/paper/2409.02244/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 109 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare](https://mlsanity.com/paper/2110.15621/) (2021)
- [Cognitive Reframing of Negative Thoughts through Human-Language Model Interaction](https://mlsanity.com/paper/2305.02466/) (2023)
- [Understanding the Benefits and Challenges of Using Large Language Model-based Conversational Agents for Mental Well-being Support](https://mlsanity.com/paper/2307.15810/) (2023)
- [Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Support](https://mlsanity.com/paper/2203.15144/) (2022)
- [Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data](https://mlsanity.com/paper/2307.14385/) (2023)
- [A Computational Framework for Behavioral Assessment of LLM Therapists](https://mlsanity.com/paper/2401.00820/) (2024)
- [Facilitating Self-Guided Mental Health Interventions Through Human-Language Model Interaction: A Case Study of Cognitive Restructuring](https://mlsanity.com/paper/2310.15461/) (2023)
- [MindfulDiary: Harnessing Large Language Model to Support Psychiatric Patients' Journaling](https://mlsanity.com/paper/2310.05231/) (2023)
- [CBT-LLM: A Chinese Large Language Model for Cognitive Behavioral Therapy-based Mental Health Question Answering](https://mlsanity.com/paper/2403.16008/) (2024)
- [Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots](https://mlsanity.com/paper/2209.03463/) (2022)
- [Helping the Helper: Supporting Peer Counselors via AI-Empowered Practice and Feedback](https://mlsanity.com/paper/2305.08982/) (2023)
- [Modeling Motivational Interviewing Strategies On An Online Peer-to-Peer Counseling Platform](https://mlsanity.com/paper/2211.05182/) (2022)
- [The Role of AI in Peer Support for Young People: A Study of Preferences for Human- and AI-Generated Responses](https://mlsanity.com/paper/2405.02711/) (2024)
- [Evaluating the Experience of LGBTQ+ People Using Large Language Model Based Chatbots for Mental Health Support](https://mlsanity.com/paper/2402.09260/) (2024)
- [Rethinking the Alignment of Psychotherapy Dialogue Generation with Motivational Interviewing Strategies](https://mlsanity.com/paper/2408.06527/) (2024)

## Built on by (most cited first, 1 of 1)

- [What Did They Mean? How LLMs Resolve Ambiguous Social Situations across Perspectives and Roles](https://mlsanity.com/paper/2604.23942/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly being used as ad hoc therapists. While prior research has found that LLMs outperform human counselors in generating single-turn empathetic responses, fewer studies have compared their behaviors across multi-turn sessions. In this study, we compare the session-level behaviors of human peer counselors with those of an LLM, both trained on the same manual to deliver multi-turn, single-session Cognitive Behavioral Therapy (CBT). Our three-phase, mixed-methods study involved: (a) an 18-month ethnography of a peer support platform, where seven counselors iteratively refined CBT prompts through 110 self-counseling sessions and 60 weekly focus groups; (b) a novel session generation method that allows direct, controlled comparison of human and LLM counselors under matched conditions---client responses were drawn from publicly available human-led CBT sessions while counselor responses were generated by a CBT-prompted LLM; and (c) expert evaluations conducted by three licensed clinical psychologists. Through data triangulation, our results show a trade-off. Human peer counselors use relational techniques to interpret subtle cues, adapt CBT to users' values and cultural contexts, and use strategies such as small talk and contextually relevant self-disclosure to build rapport and guide the session, but often at the expense of session structure and therapeutic focus. LLM counselors, on the other hand, demonstrate greater methodological adherence to CBT techniques, but struggle to sustain turn-taking, frequently fail to distinguish between clinically important and trivial content, and are more prone to lecturing and imposing solutions. LLM counselors also tend to produce ``deceptive empathy'', excessively anthropomorphic responses that can inflate user expectations of genuine human care.