# MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models

arXiv: https://arxiv.org/abs/2312.12806 · HTML: https://mlsanity.com/paper/2312.12806/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 20 Dec 2023.

## Builds on (most cited first, 15 of 15)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge](https://mlsanity.com/paper/2304.06975/) (2023)
- [DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task](https://mlsanity.com/paper/2304.01097/) (2023)
- [Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset](https://mlsanity.com/paper/2306.03030/) (2023)
- [PMC-LLaMA: Towards Building Open-source Language Models for Medicine](https://mlsanity.com/paper/2304.14454/) (2023)
- [BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT](https://mlsanity.com/paper/2310.15896/) (2023)
- [MedGPTEval: A Dataset and Benchmark to Evaluate Responses of Large Language Models in Medicine](https://mlsanity.com/paper/2305.07340/) (2023)

## Built on by (most cited first, 30 of 31)

- [RareBench: Can LLMs Serve as Rare Diseases Specialists?](https://mlsanity.com/paper/2402.06341/) (2024)
- [A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directions](https://mlsanity.com/paper/2406.03712/) (2024)
- [ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?](https://mlsanity.com/paper/2411.06469/) (2024)
- [ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real World](https://mlsanity.com/paper/2406.13890/) (2024)
- [A Survey of Conversational Search](https://mlsanity.com/paper/2410.15576/) (2024)
- [A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry](https://mlsanity.com/paper/2404.15777/) (2024)
- [TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine](https://mlsanity.com/paper/2406.01126/) (2024)
- [Medchain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence](https://mlsanity.com/paper/2412.01605/) (2024)
- [LiveMedBench: A Contamination-Free Medical Benchmark for LLMs with Automated Rubric Evaluation](https://mlsanity.com/paper/2602.10367/) (2026)
- [MAP: Evaluation and Multi-Agent Enhancement of Large Language Models for Inpatient Pathways](https://mlsanity.com/paper/2503.13205/) (2025)
- [It is Too Many Options: Pitfalls of Multiple-Choice Questions in Generative AI and Medical Education](https://mlsanity.com/paper/2503.13508/) (2025)
- [Enhancing the Medical Context-Awareness Ability of LLMs via Multifaceted Self-Refinement Learning](https://mlsanity.com/paper/2511.10067/) (2025)
- [Measuring Epistemic Resilience of LLMs Under Misleading Medical Context](https://mlsanity.com/paper/2606.12291/) (2026)
- [Are Large Language Models True Healthcare Jacks-of-All-Trades? Benchmarking Across Health Professions Beyond Physician Exams](https://mlsanity.com/paper/2406.11328/) (2024)
- [DALL-M: Context-Aware Clinical Data Augmentation with LLMs](https://mlsanity.com/paper/2407.08227/) (2024)
- [OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmology](https://mlsanity.com/paper/2502.01243/) (2025)
- [Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmap](https://mlsanity.com/paper/2508.18646/) (2025)
- [LingLanMiDian: Systematic Evaluation of LLMs on TCM Knowledge and Clinical Reasoning](https://mlsanity.com/paper/2602.01779/) (2026)
- [Toward Workflow-Aware Benchmarking for Healthcare NLP Agents](https://mlsanity.com/paper/2609.00296/) (2026)
- [Advancing Chinese biomedical text mining with community challenges](https://mlsanity.com/paper/2403.04261/) (2024)
- [Towards Automatic Continual Learning: A Self-Adaptive Framework for Continual Instruction Tuning](https://mlsanity.com/paper/2503.15924/) (2025)
- [NurValues: Real-World Nursing Values Evaluation for Large Language Models in Clinical Context](https://mlsanity.com/paper/2505.08734/) (2025)
- [Inclusion Arena: An Open Platform for Evaluating Large Foundation Models with Real-World Apps](https://mlsanity.com/paper/2508.11452/) (2025)
- [A benchmark dataset for evaluating Syndrome Differentiation and Treatment in large language models](https://mlsanity.com/paper/2512.02816/) (2025)
- [Deep Learning-based Method for Expressing Knowledge Boundary of Black-Box LLM](https://mlsanity.com/paper/2602.10801/) (2026)
- [LitBench: A Graph-Centric Large Language Model Benchmarking Tool For Literature Tasks](https://mlsanity.com/paper/2603.00051/) (2026)
- [Leveraging Computerized Adaptive Testing for Cost-effective Evaluation of Large Language Models in Medical Benchmarking](https://mlsanity.com/paper/2603.23506/) (2026)
- [MedThink: Enhancing Diagnostic Accuracy in Small Models via Teacher-Guided Reasoning Correction](https://mlsanity.com/paper/2605.08094/) (2026)
- [Didactic knowledge or Clinical Cases? How Data Types Shape Medical Large Language Models](https://mlsanity.com/paper/2609.22161/) (2026)
- [Clinician use of language models diverges from how the models are evaluated](https://mlsanity.com/paper/2610.11069/) (2026)

## Abstract (arXiv)

The emergence of various medical large language models (LLMs) in the medical domain has highlighted the need for unified evaluation standards, as manual evaluation of LLMs proves to be time-consuming and labor-intensive. To address this issue, we introduce MedBench, a comprehensive benchmark for the Chinese medical domain, comprising 40,041 questions sourced from authentic examination exercises and medical reports of diverse branches of medicine. In particular, this benchmark is composed of four key components: the Chinese Medical Licensing Examination, the Resident Standardization Training Examination, the Doctor In-Charge Qualification Examination, and real-world clinic cases encompassing examinations, diagnoses, and treatments. MedBench replicates the educational progression and clinical practice experiences of doctors in Mainland China, thereby establishing itself as a credible benchmark for assessing the mastery of knowledge and reasoning abilities in medical language learning models. We perform extensive experiments and conduct an in-depth analysis from diverse perspectives, which culminate in the following findings: (1) Chinese medical LLMs underperform on this benchmark, highlighting the need for significant advances in clinical knowledge and diagnostic precision. (2) Several general-domain LLMs surprisingly possess considerable medical knowledge. These findings elucidate both the capabilities and limitations of LLMs within the context of MedBench, with the ultimate goal of aiding the medical research community.