mlsanityData to 7 Oct 2026

CMB: A Comprehensive Medical Benchmark in Chinese

arXiv 2308.08833 ↗First posted 17 Aug 2023cs.CL cs.AIcited by 113

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) provide a possibility to make a great breakthrough in medicine. The establishment of a standardized medical benchmark becomes a fundamental cornerstone to measure progression. However, medical environments in different regions have their local characteristics, e.g., the ubiquity and significance of traditional Chinese medicine within China. Therefore, merely translating English-based medical evaluation may result in textitcontextual incongruities to a local region. To solve the issue, we propose a localized medical benchmark called CMB, a Comprehensive Medical Benchmark in Chinese, designed and rooted entirely within the native Chinese linguistic and cultural framework. While traditional Chinese medicine is integral to this evaluation, it does not constitute its entirety. Using this benchmark, we have evaluated several prominent large-scale LLMs, including ChatGPT, GPT-4, dedicated Chinese LLMs, and LLMs specialized in the medical domain. We hope this benchmark provide first-hand experience in existing LLMs for medicine and also facilitate the widespread adoption and enhancement of medical LLMs within China. Our data and code are publicly available at https://github.com/FreedomIntelligence/CMB.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Builds on 34 of its 54 references that are in the map. First posted on arXiv 17 Aug 2023.

Builds on

34 of 54 references.

References · 34

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  12. 2021Measuring Coding Challenge Competence With APPScited by 879
22 more
  1. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  2. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  3. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  4. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  5. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  6. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 211
  7. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  8. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  9. 2023HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledgecited by 161
  10. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 134
  11. 2019A Question-Entailment Approach to Question Answeringcited by 133
  12. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 130
  13. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  14. 2021CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmarkcited by 99
  15. 2023Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Datasetcited by 90
  16. 2021Can Language Models be Biomedical Knowledge Bases?cited by 83
  17. 2022Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligencecited by 70
  18. 2023ARB: Advanced Reasoning Benchmark for Large Language Modelscited by 60
  19. 2023Huatuo-26M, a Large-scale Chinese Medical QA Datasetcited by 46
  20. 2018CliCR: A Dataset of Clinical Case Reports for Machine Reading Comprehensioncited by 34
  21. 2023ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examinationcited by 10
  22. 2023IvyGPT: InteractiVe Chinese pathwaY language model in medical domaincited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 41; 2025: 45; 2026: 19.

113 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  4. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  5. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  6. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
  7. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  8. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 91
  9. 2023DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultationcited by 89
  10. 2025LLM Agents for Education: Advances and Applicationscited by 59
  11. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 41
  12. 2024Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B Peoplecited by 38
38 more
  1. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  2. 2024ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?cited by 31
  3. 2024ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real Worldcited by 28
  4. 2024TC-RAG:Turing-Complete RAG's Case study on Medical LLM Systemscited by 28
  5. 2025Synergizing RAG and Reasoning: A Systematic Reviewcited by 27
  6. 2024PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluationcited by 26
  7. 2023HyKGE: A Hypothesis Knowledge Graph Enhanced Framework for Accurate and Reliable Medical LLMs Responsescited by 25
  8. 2024Training on the Benchmark Is Not All You Needcited by 24
  9. 2024MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Modelscited by 21
  10. 2024KnowPO: Knowledge-aware Preference Optimization for Controllable Knowledge Selection in Retrieval-Augmented Language Modelscited by 20
  11. 2025FineMedLM-o1: Enhancing Medical Knowledge Reasoning Ability of LLM from Supervised Fine-Tuning to Test-Time Trainingcited by 19
  12. 2025DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Modelscited by 19
  13. 2023Knowledge-tuning Large Language Models with Structured Medical Knowledge Bases for Reliable Response Generation in Chinesecited by 18
  14. 2024ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Modelscited by 18
  15. 2024A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industrycited by 17
  16. 2024Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Expertscited by 17
  17. 2024medIKAL: Integrating Knowledge Graphs as Assistants of LLMs for Enhanced Clinical Diagnosis on EMRscited by 16
  18. 2024Medchain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequencecited by 15
  19. 2025SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Modelscited by 15
  20. 2024E-EVAL: A Comprehensive Chinese K-12 Education Evaluation Benchmark for Large Language Modelscited by 13
  21. 2024M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Modelscited by 13
  22. 2025Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applicationscited by 13
  23. 2025ProMed: Shapley Information Gain Guided Reinforcement Learning for Proactive Medical LLMscited by 12
  24. 2024BianCang: A Traditional Chinese Medicine Large Language Modelcited by 11
  25. 2025Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMscited by 10
  26. 2024CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarioscited by 9
  27. 2025MAP: Evaluation and Multi-Agent Enhancement of Large Language Models for Inpatient Pathwayscited by 9
  28. 2026LiveMedBench: A Contamination-Free Medical Benchmark for LLMs with Automated Rubric Evaluationcited by 9
  29. 2024AutoMIR: Effective Zero-Shot Medical Information Retrieval without Relevance Labelscited by 8
  30. 2025The Multi-Round Diagnostic RAG Framework for Emulating Clinical Reasoningcited by 8
  31. 2026EvoClinician: A Self-Evolving Agent for Multi-Turn Medical Diagnosis via Test-Time Evolutionary Learningcited by 8
  32. 2024MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Expertscited by 7
  33. 2025MedKGI: Iterative Differential Diagnosis with Medical Knowledge Graphs and Information-Guided Inquiringcited by 7
  34. 2024Honest AI: Fine-Tuning "Small" Language Models to Say "I Don't Know", and Reducing Hallucination in RAGcited by 6
  35. 2025Add-One-In: Incremental Sample Selection for Large Language Models via a Choice-Based Greedy Paradigmcited by 6
  36. 2023OpsEval: A Comprehensive IT Operations Benchmark Suite for Large Language Modelscited by 5
  37. 2024Medical Dialogue: A Survey of Categories, Methods, Evaluation and Challengescited by 5
  38. 2024TCMD: A Traditional Chinese Medicine QA Dataset for Evaluating Large Language Modelscited by 5

50 most cited of 113 All 113 (JSON)