# An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregivers

arXiv: https://arxiv.org/abs/2107.13115 · HTML: https://mlsanity.com/paper/2107.13115/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2021.

## Built on by (most cited first, 15 of 15)

- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [Multilingual Language Models are not Multicultural: A Case Study in Emotion](https://mlsanity.com/paper/2307.01370/) (2023)
- [MindfulDiary: Harnessing Large Language Model to Support Psychiatric Patients' Journaling](https://mlsanity.com/paper/2310.05231/) (2023)
- [Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data](https://mlsanity.com/paper/2301.05843/) (2023)
- [Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personas](https://mlsanity.com/paper/2408.06929/) (2024)
- [Understanding the Impact of Long-Term Memory on Self-Disclosure with Large Language Model-Driven Chatbots for Public Health Intervention](https://mlsanity.com/paper/2402.11353/) (2024)
- [Storyfier: Exploring Vocabulary Learning Support with Text Generation Models](https://mlsanity.com/paper/2308.03864/) (2023)
- [Toward Large Language Models as a Therapeutic Tool: Comparing Prompting Techniques to Improve GPT-Delivered Problem-Solving Therapy](https://mlsanity.com/paper/2409.00112/) (2024)
- [FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Health](https://mlsanity.com/paper/2509.14275/) (2025)
- [From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysis](https://mlsanity.com/paper/2501.06101/) (2025)
- [Rethinking AI Cultural Alignment](https://mlsanity.com/paper/2501.07751/) (2025)
- [SeSaMe: A Framework to Simulate Self-Reported Ground Truth for Mental Health Sensing Studies](https://mlsanity.com/paper/2403.17219/) (2024)
- [A Risk Taxonomy and Reflection Tool for Large Language Model Adoption in Public Health](https://mlsanity.com/paper/2411.02594/) (2024)
- [Large Language Model-Powered Conversational Agent Delivering Problem-Solving Therapy (PST) for Family Caregivers: Enhancing Empathy and Therapeutic Alliance Using In-Context Learning](https://mlsanity.com/paper/2506.11376/) (2025)
- [Structure Matters: Evaluating Multi-Agents Orchestration in Generative Therapeutic Chatbots](https://mlsanity.com/paper/2603.00774/) (2026)

## Abstract (arXiv)

With the advent of off-the-shelf intelligent home products and broader internet adoption, researchers increasingly explore smart computing applications that provide easier access to health and wellness resources. AI-based systems like chatbots have the potential to provide services that could provide mental health support. However, existing therapy chatbots are often retrieval-based, requiring users to respond with a constrained set of answers, which may not be appropriate given that such pre-determined inquiries may not reflect each patient's unique circumstances. Generative-based approaches, such as the OpenAI GPT models, could allow for more dynamic conversations in therapy chatbot contexts than previous approaches. To investigate the generative-based model's potential in therapy chatbot contexts, we built a chatbot using the GPT-2 model. We fine-tuned it with 306 therapy session transcripts between family caregivers of individuals with dementia and therapists conducting Problem Solving Therapy. We then evaluated the model's pre-trained and the fine-tuned model in terms of basic qualities using three meta-information measurements: the proportion of non-word outputs, the length of response, and sentiment components. Results showed that: (1) the fine-tuned model created more non-word outputs than the pre-trained model; (2) the fine-tuned model generated outputs whose length was more similar to that of the therapists compared to the pre-trained model; (3) both the pre-trained model and fine-tuned model were likely to generate more negative and fewer positive outputs than the therapists. We discuss potential reasons for the problem, the implications, and solutions for developing therapy chatbots and call for investigations of the AI-based system application.