# A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets

arXiv: https://arxiv.org/abs/2204.08997 · HTML: https://mlsanity.com/paper/2204.08997/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 56 references that are in the map.
- First posted on arXiv 19 Apr 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [MedDialog: Two Large-scale Medical Dialogue Datasets](https://mlsanity.com/paper/2004.03329/) (2020)
- [Recurrent Neural Network for Text Classification with Multi-Task Learning](https://mlsanity.com/paper/1605.05101/) (2016)
- [CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2106.08087/) (2021)
- [Chinese NER Using Lattice LSTM](https://mlsanity.com/paper/1805.02023/) (2018)
- [End-to-End Knowledge-Routed Relational Dialogue System for Automatic Diagnosis](https://mlsanity.com/paper/1901.10623/) (2019)
- [Dr. Summarize: Global Summarization of Medical Dialogue by Exploiting Local Structures](https://mlsanity.com/paper/2009.08666/) (2020)
- [MedDG: An Entity-Centric Medical Consultation Dataset for Entity-Aware Medical Dialogue Generation](https://mlsanity.com/paper/2010.07497/) (2020)
- [Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition](https://mlsanity.com/paper/2010.03746/) (2020)
- [Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter](https://mlsanity.com/paper/2105.07148/) (2021)
- [ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation](https://mlsanity.com/paper/2104.08006/) (2021)
- [Semi-Supervised Variational Reasoning for Medical Dialogue Generation](https://mlsanity.com/paper/2105.06071/) (2021)
- [Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation](https://mlsanity.com/paper/2012.11988/) (2020)
- [Extracting Symptoms and their Status from Clinical Conversations](https://mlsanity.com/paper/1906.02239/) (2019)
- [Drug Package Recommendation via Interaction-aware Graph Induction](https://mlsanity.com/paper/2102.03577/) (2021)

## Built on by (most cited first, 30 of 45)

- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator](https://mlsanity.com/paper/2402.09742/) (2024)
- [DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultation](https://mlsanity.com/paper/2308.14346/) (2023)
- [BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT](https://mlsanity.com/paper/2310.15896/) (2023)
- [PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain](https://mlsanity.com/paper/2310.14151/) (2023)
- [Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator](https://mlsanity.com/paper/2403.08495/) (2024)
- [Real-World Doctor Agent with Proactive Consultation through Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2505.19630/) (2025)
- [MediTOD: An English Dialogue Dataset for Medical History Taking with Comprehensive Annotations](https://mlsanity.com/paper/2410.14204/) (2024)
- [MC-CoT: A Modular Collaborative CoT Framework for Zero-shot Medical-VQA with LLM and MLLM Integration](https://mlsanity.com/paper/2410.04521/) (2024)
- [Health-LLM: Personalized Retrieval-Augmented Disease Prediction System](https://mlsanity.com/paper/2402.00746/) (2024)
- [A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry](https://mlsanity.com/paper/2404.15777/) (2024)
- [Large Language Model Benchmarks in Medical Tasks](https://mlsanity.com/paper/2410.21348/) (2024)
- [3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark](https://mlsanity.com/paper/2504.13861/) (2025)
- [ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark](https://mlsanity.com/paper/2507.05727/) (2025)
- [Overview of the PromptCBLUE Shared Task in CHIP2023](https://mlsanity.com/paper/2312.17522/) (2023)
- [MidMed: Towards Mixed-Type Dialogues for Medical Consultation](https://mlsanity.com/paper/2306.02923/) (2023)
- [Towards Online Multi-Modal Social Interaction Understanding](https://mlsanity.com/paper/2503.19851/) (2025)
- [InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training](https://mlsanity.com/paper/2510.15859/) (2025)
- [DxFormer: A Decoupled Automatic Diagnostic System Based on Decoder-Encoder Transformer with Dense Symptom Representations](https://mlsanity.com/paper/2205.03755/) (2022)
- [Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback](https://mlsanity.com/paper/2401.05695/) (2024)
- [PlugMed: Improving Specificity in Patient-Centered Medical Dialogue Generation using In-Context Learning](https://mlsanity.com/paper/2305.11508/) (2023)
- [PIORS: Personalized Intelligent Outpatient Reception based on Large Language Model with Multi-Agents Medical Scenario Simulation](https://mlsanity.com/paper/2411.13902/) (2024)
- [KNSE: A Knowledge-aware Natural Language Inference Framework for Dialogue Symptom Status Recognition](https://mlsanity.com/paper/2305.16833/) (2023)
- [Medical Dialogue: A Survey of Categories, Methods, Evaluation and Challenges](https://mlsanity.com/paper/2405.10630/) (2024)
- [CauESC: A Causal Aware Model for Emotional Support Conversation](https://mlsanity.com/paper/2401.17755/) (2024)
- [Omni-MMSI: Toward Identity-attributed Social Interaction Understanding](https://mlsanity.com/paper/2604.00267/) (2026)
- [Conversational Disease Diagnosis via External Planner-Controlled Large Language Models](https://mlsanity.com/paper/2404.04292/) (2024)
- [Improving Interactive Diagnostic Ability of a Large Language Model Agent Through Clinical Experience Learning](https://mlsanity.com/paper/2503.16463/) (2025)
- [Beyond Idealized Patients: Evaluating LLMs under Challenging Patient Behaviors in Medical Consultations](https://mlsanity.com/paper/2603.29373/) (2026)
- [Medical Dialogue Generation via Intuitive-then-Analytical Differential Diagnosis](https://mlsanity.com/paper/2401.06541/) (2024)

## Abstract (arXiv)

In recent years, interest has arisen in using machine learning to improve the efficiency of automatic medical consultation and enhance patient experience. In this article, we propose two frameworks to support automatic medical consultation, namely doctor-patient dialogue understanding and task-oriented interaction. We create a new large medical dialogue dataset with multi-level finegrained annotations and establish five independent tasks, including named entity recognition, dialogue act classification, symptom label inference, medical report generation and diagnosis-oriented dialogue policy. We report a set of benchmark results for each task, which shows the usability of the dataset and sets a baseline for future studies. Both code and data is available from https://github.com/lemuria-wchen/imcs21.