mlsanityData to 7 Oct 2026

Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions

arXiv 2405.16402 ↗First posted 26 May 2024cs.CL cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

The integration of Large Language Models (LLMs) into the healthcare domain has the potential to significantly enhance patient care and support through the development of empathetic, patient-facing chatbots. This study investigates an intriguing question Can ChatGPT respond with a greater degree of empathy than those typically offered by physicians? To answer this question, we collect a de-identified dataset of patient messages and physician responses from Mayo Clinic and generate alternative replies using ChatGPT. Our analyses incorporate novel empathy ranking evaluation (EMRank) involving both automated metrics and human assessments to gauge the empathy level of responses. Our findings indicate that LLM-powered chatbots have the potential to surpass human physicians in delivering empathetic communication, suggesting a promising avenue for enhancing patient care and reducing professional burnout. The study not only highlights the importance of empathy in patient interactions but also proposes a set of effective automatic empathy ranking metrics, paving the way for the broader adoption of LLMs in healthcare.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 25 of its 59 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

25 of 59 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  8. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  9. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  10. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  11. 2020A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Supportcited by 182
  12. 2023SoulChat: Improving LLMs' Empathy, Listening, and Comfort Abilities through Fine-tuning with Multi-turn Empathy Conversationscited by 109
13 more
  1. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  2. 2023Dr.ICL: Demonstration-Retrieved In-context Learningcited by 71
  3. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  4. 2023Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Datacited by 66
  5. 2021Socially Responsible AI Algorithms: Issues, Purposes, and Challengescited by 56
  6. 2020A Taxonomy of Empathetic Response Intents in Human Social Conversationscited by 51
  7. 2021Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causescited by 43
  8. 2024If in a Crowdsourced Data Annotation Pipeline, a GPT-4cited by 38
  9. 2022Improving Biomedical Information Retrieval with Neural Retrieverscited by 20
  10. 2022In-BoXBART: Get Instructions into Biomedical Multi-Task Learningcited by 16
  11. 2022Improving Contextual Coherence in Variational Personalized and Empathetic Dialogue Agentscited by 14
  12. 2023MDDial: A Multi-turn Differential Diagnosis Dialogue Dataset with Reliability Evaluationcited by 10
  13. 2023Artificial Empathy Classification: A Survey of Deep Learning Techniques, Datasets, and Evaluation Scalescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 4.

13 papers.

Sort:

By area:

  1. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  2. 2024Script-Strategy Aligned Generation: Aligning LLMs with Expert-Crafted Dialogue Scripts and Therapeutic Strategies for Psychotherapycited by 7
  3. 2025Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)cited by 7
  4. 2025RECAP: Transparent Inference-Time Emotion Alignment for Medical Dialogue Systemscited by 3
  5. 2024The use of GPT-4o and Other Large Language Models for the Improvement and Design of Self-Assessment Scales for Measurement of Interpersonal Communication Skillscited by 1
  6. 2024A Multi-Agent Dual Dialogue System to Support Mental Health Care Providerscited by 1
  7. 2025SocialLM: Social Signal Processing of Patient-Provider Communication using LLMs and Contextual Aggregationcited by 1
  8. 2025Empathy by Design: Aligning Large Language Models for Healthcare Dialoguecited by 1
  9. 2025Context-Emotion Aware Therapeutic Dialogue Generation: A Multi-component Reinforcement Learning Approach to Language Models for Mental Health Supportcited by 0
  10. 2026From Generation to Collaboration: Using LLMs to Edit for Empathy in Healthcarecited by 0
  11. 2026LLMs Should Incorporate Explicit Mechanisms for Human Empathycited by 0
  12. 2026Can "AI" Be a Doctor? A Study of Empathy, Readability, and Alignment in Clinical LLMscited by 0
1 more
  1. 2026Appropriateness of Empathy in AI: A Signal-Cost Perspectivecited by 0