# LLM-empowered Chatbots for Psychiatrist and Patient Simulation: Application and Evaluation

arXiv: https://arxiv.org/abs/2305.13614 · HTML: https://mlsanity.com/paper/2305.13614/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 13 of 13)

- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [DDXPlus: A New Dataset For Automatic Medical Diagnosis](https://mlsanity.com/paper/2205.09148/) (2022)
- [CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation](https://mlsanity.com/paper/2109.05729/) (2021)
- [End-to-End Knowledge-Routed Relational Dialogue System for Automatic Diagnosis](https://mlsanity.com/paper/1901.10623/) (2019)
- [Is ChatGPT Equipped with Emotional Dialogue Capabilities?](https://mlsanity.com/paper/2304.09582/) (2023)
- [Evaluation of ChatGPT for NLP-based Mental Health Applications](https://mlsanity.com/paper/2303.15727/) (2023)
- [Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data](https://mlsanity.com/paper/2301.05843/) (2023)
- [D4: a Chinese Dialogue Dataset for Depression-Diagnosis-Oriented Chat](https://mlsanity.com/paper/2205.11764/) (2022)
- [Transferable Dialogue Systems and User Simulators](https://mlsanity.com/paper/2107.11904/) (2021)
- [Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation](https://mlsanity.com/paper/2012.11988/) (2020)
- [Symptom Identification for Interpretable Detection of Multiple Mental Disorders](https://mlsanity.com/paper/2205.11308/) (2022)
- [Chatbots for Mental Health Support: Exploring the Impact of Emohaa on Reducing Mental Distress in China](https://mlsanity.com/paper/2209.10183/) (2022)
- [Read, Diagnose and Chat: Towards Explainable and Interactive LLMs-Augmented Depression Detection in Social Media](https://mlsanity.com/paper/2305.05138/) (2023)

## Built on by (most cited first, 30 of 34)

- [Hello Again! LLM-powered Personalized Agent for Long-term Dialogue](https://mlsanity.com/paper/2406.05925/) (2024)
- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator](https://mlsanity.com/paper/2403.08495/) (2024)
- [ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Events](https://mlsanity.com/paper/2309.12244/) (2023)
- [Enhancing Psychological Counseling with Large Language Model: A Multifaceted Decision-Support System for Non-Professionals](https://mlsanity.com/paper/2308.15192/) (2023)
- [Leveraging Large Language Model as Simulated Patients for Clinical Education](https://mlsanity.com/paper/2404.13066/) (2024)
- [User Modeling in the Era of Large Language Models: Current Research and Future Directions](https://mlsanity.com/paper/2312.11518/) (2023)
- [Technological folie `a deux: Feedback Loops Between AI Chatbots and Mental Illness](https://mlsanity.com/paper/2507.19218/) (2025)
- [Large Language Model for Mental Health: A Systematic Review](https://mlsanity.com/paper/2403.15401/) (2024)
- [Prompt engineering paradigms for medical applications: scoping review and recommendations for better practices](https://mlsanity.com/paper/2405.01249/) (2024)
- [An Automatic Evaluation Framework for Multi-turn Medical Consultations Capabilities of Large Language Models](https://mlsanity.com/paper/2309.02077/) (2023)
- [Preference Learning Unlocks LLMs' Psycho-Counseling Skills](https://mlsanity.com/paper/2502.19731/) (2025)
- [A Survey of Scaling in Large Language Model Reasoning](https://mlsanity.com/paper/2504.02181/) (2025)
- [Supervised Learning and Large Language Model Benchmarks on Mental Health Datasets: Cognitive Distortions and Suicidal Risks in Chinese Social Media](https://mlsanity.com/paper/2309.03564/) (2023)
- [Towards a Psychological Generalist AI: A Survey of Current Applications of Large Language Models and Future Prospects](https://mlsanity.com/paper/2312.04578/) (2023)
- [A Comprehensive Evaluation of Large Language Models on Mental Illnesses](https://mlsanity.com/paper/2409.15687/) (2024)
- [Hybrid Training Approaches for LLMs: Leveraging Real and Synthetic Data to Enhance Model Performance in Domain-Specific Applications](https://mlsanity.com/paper/2410.09168/) (2024)
- [AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs](https://mlsanity.com/paper/2410.19692/) (2024)
- [Modeling Challenging Patient Interactions: LLMs for Medical Communication Training](https://mlsanity.com/paper/2503.22250/) (2025)
- [Harnessing Large Language Models Over Transformer Models for Detecting Bengali Depressive Social Media Text: A Comprehensive Study](https://mlsanity.com/paper/2401.07310/) (2024)
- [Large Language Multimodal Models for 5-Year Chronic Disease Cohort Prediction Using EHR Data](https://mlsanity.com/paper/2403.04785/) (2024)
- [Toward Large Language Models as a Therapeutic Tool: Comparing Prompting Techniques to Improve GPT-Delivered Problem-Solving Therapy](https://mlsanity.com/paper/2409.00112/) (2024)
- [MADP: Multi-Agent Deductive Planning for Enhanced Cognitive-Behavioral Mental Health Question Answer](https://mlsanity.com/paper/2501.15826/) (2025)
- [TRUST: An LLM-Based Dialogue System for Trauma Understanding and Structured Assessments](https://mlsanity.com/paper/2504.21851/) (2025)
- [MedDiT: A Knowledge-Controlled Diffusion Transformer Framework for Dynamic Medical Image Generation in Virtual Simulated Patient](https://mlsanity.com/paper/2408.12236/) (2024)
- [Mixed Chain-of-Psychotherapies for Emotional Support Chatbot](https://mlsanity.com/paper/2409.19533/) (2024)
- [Increasing happiness through conversations with artificial intelligence](https://mlsanity.com/paper/2504.02091/) (2025)
- [DS@GT at eRisk 2025: From prompts to predictions, benchmarking early depression detection with conversational agent based assessments and temporal attention models](https://mlsanity.com/paper/2507.10958/) (2025)
- [H-DDx: A Hierarchical Evaluation Framework for Differential Diagnosis](https://mlsanity.com/paper/2510.03700/) (2025)
- [From Medical Records to Diagnostic Dialogues: A Clinical-Grounded Approach and Dataset for Psychiatric Comorbidity](https://mlsanity.com/paper/2510.25232/) (2025)

## Abstract (arXiv)

Empowering chatbots in the field of mental health is receiving increasing amount of attention, while there still lacks exploration in developing and evaluating chatbots in psychiatric outpatient scenarios. In this work, we focus on exploring the potential of ChatGPT in powering chatbots for psychiatrist and patient simulation. We collaborate with psychiatrists to identify objectives and iteratively develop the dialogue system to closely align with real-world scenarios. In the evaluation experiments, we recruit real psychiatrists and patients to engage in diagnostic conversations with the chatbots, collecting their ratings for assessment. Our findings demonstrate the feasibility of using ChatGPT-powered chatbots in psychiatric scenarios and explore the impact of prompt designs on chatbot behavior and user experience.