mlsanityData to 14 Oct 2026

CMQCIC-Bench: A Chinese Benchmark for Evaluating Large Language Models in Medical Quality Control Indicator Calculation

arXiv 2502.11703 ↗First posted 17 Feb 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Medical quality control indicators are essential to assess the qualifications of healthcare institutions for medical services. With the impressive performance of large language models (LLMs) like GPT-4 in the medical field, leveraging these technologies for the Medical Quality Control Indicator Calculation (MQCIC) presents a promising approach. In this work, (1) we introduce a real-world task MQCIC and propose an open-source Chinese electronic medical records (EMRs)-based dataset (CMQCIC-Bench) comprising 785 instances and 76 indicators. (2) We propose a semi-automatic method to enhance the rule representation. Then we propose the Clinical Facts-based Inferential Rule (CF-IR) method that disentangles the clinical fact verification and inferential rule reasoning actions. (3) We conduct comprehensive experiments on 20 representative LLMs, covering general and medical models. Our findings reveal that CF-IR outperforms Chain-of-Thought methods in MQCIC tasks. (4) We conduct an error analysis and investigate the capabilities of clinical fact verification and inferential rule reasoning, providing insights to improve performance in the MQCIC further. The dataset and code is available in this repository https://github.com/YuY-2001/C-MQCIC.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 44 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

26 of 44 references.

References · 26

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  8. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  9. 2024Faithful Logical Reasoning via Symbolic Chain-of-Thoughtcited by 100
  10. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 92
  11. 2024MedCalc-Bench: Evaluating Large Language Models for Medical Calculationscited by 70
  12. 2024Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B Peoplecited by 38
14 more
  1. 2023Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulationcited by 30
  2. 2024Beyond Instruction Following: Evaluating Inferential Rule Following of Large Language Modelscited by 23
  3. 2024Divide-or-Conquer? Which Part Should You Distill Your LLM?cited by 17
  4. 2024A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industrycited by 17
  5. 2024Symbolic Working Memory Enhances Language Models for Complex Rule Applicationcited by 17
  6. 2024Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Expertscited by 17
  7. 2025VeriFact: Verifying Facts in LLM-Generated Clinical Text with Electronic Health Recordscited by 17
  8. 2024DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Modelscited by 14
  9. 2024Can LLMs Reason in the Wild with Programs?cited by 13
  10. 2024Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedbackcited by 9
  11. 2024CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarioscited by 9
  12. 2024M-QALM: A Benchmark to Assess Clinical Reading Comprehension and Knowledge Recall in Large Language Models via Question Answeringcited by 3
  13. 2023ExpNote: Black-box Large Language Models are Better Task Solvers with Experience Notebookcited by 2
  14. 2024MSDiagnosis: A Benchmark for Evaluating Large Language Models in Multi-Step Clinical Diagnosiscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
  2. 2025Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategies
  3. 2025MedFact: Benchmarking the Fact-Checking Capabilities of Large Language Models on Chinese Medical Texts
  4. 2025MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine
  5. 2025Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models
  6. 2026Medical Reasoning with Large Language Models: A Survey and MR-Bench
  7. 2025Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases
  8. 2023Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset