# Benchmarking Large Language Models on Communicative Medical Coaching: a Novel System and Dataset

arXiv: https://arxiv.org/abs/2402.05547 · HTML: https://mlsanity.com/paper/2402.05547/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 34 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data](https://mlsanity.com/paper/2302.12822/) (2023)
- [CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2106.08087/) (2021)
- [MidMed: Towards Mixed-Type Dialogues for Medical Consultation](https://mlsanity.com/paper/2306.02923/) (2023)
- [Deep Graph Random Process for Relational-Thinking-Based Speech Recognition](https://mlsanity.com/paper/2007.02126/) (2020)
- [STRODE: Stochastic Boundary Ordinary Differential Equation](https://mlsanity.com/paper/2107.08273/) (2021)
- [Distantly supervised end-to-end medical entity extraction from electronic health records with human-level quality](https://mlsanity.com/paper/2201.10463/) (2022)

## Built on by (most cited first, 2 of 2)

- [A Survey of LLM-based Agents in Medicine: How far are we from Baymax?](https://mlsanity.com/paper/2502.11211/) (2025)
- [Constructing and Evaluating Clinical Reasoning Trajectories for Medical Agent](https://mlsanity.com/paper/2609.05090/) (2026)

## Abstract (arXiv)

Traditional applications of natural language processing (NLP) in healthcare have predominantly focused on patient-centered services, enhancing patient interactions and care delivery, such as through medical dialogue systems. However, the potential of NLP to benefit inexperienced doctors, particularly in areas such as communicative medical coaching, remains largely unexplored. We introduce "ChatCoach", a human-AI cooperative framework designed to assist medical learners in practicing their communication skills during patient consultations. ChatCoach (Our data and code are available online: https://github.com/zerowst/Chatcoach)differentiates itself from conventional dialogue systems by offering a simulated environment where medical learners can practice dialogues with a patient agent, while a coach agent provides immediate, structured feedback. This is facilitated by our proposed Generalized Chain-of-Thought (GCoT) approach, which fosters the generation of structured feedback and enhances the utilization of external knowledge sources. Additionally, we have developed a dataset specifically for evaluating Large Language Models (LLMs) within the ChatCoach framework on communicative medical coaching tasks. Our empirical results validate the effectiveness of ChatCoach.