# CMQCIC-Bench: A Chinese Benchmark for Evaluating Large Language Models in Medical Quality Control Indicator Calculation

arXiv: https://arxiv.org/abs/2502.11703 · HTML: https://mlsanity.com/paper/2502.11703/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Faithful Logical Reasoning via Symbolic Chain-of-Thought](https://mlsanity.com/paper/2405.18357/) (2024)
- [HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs](https://mlsanity.com/paper/2311.09774/) (2023)
- [MedCalc-Bench: Evaluating Large Language Models for Medical Calculations](https://mlsanity.com/paper/2406.12036/) (2024)
- [Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B People](https://mlsanity.com/paper/2403.03640/) (2024)
- [Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulation](https://mlsanity.com/paper/2310.15746/) (2023)
- [Beyond Instruction Following: Evaluating Inferential Rule Following of Large Language Models](https://mlsanity.com/paper/2407.08440/) (2024)
- [Divide-or-Conquer? Which Part Should You Distill Your LLM?](https://mlsanity.com/paper/2402.15000/) (2024)
- [A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry](https://mlsanity.com/paper/2404.15777/) (2024)
- [Symbolic Working Memory Enhances Language Models for Complex Rule Application](https://mlsanity.com/paper/2408.13654/) (2024)
- [Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts](https://mlsanity.com/paper/2410.10626/) (2024)
- [VeriFact: Verifying Facts in LLM-Generated Clinical Text with Electronic Health Records](https://mlsanity.com/paper/2501.16672/) (2025)
- [DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models](https://mlsanity.com/paper/2408.01933/) (2024)
- [Can LLMs Reason in the Wild with Programs?](https://mlsanity.com/paper/2406.13764/) (2024)
- [Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback](https://mlsanity.com/paper/2401.05695/) (2024)
- [CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios](https://mlsanity.com/paper/2410.03502/) (2024)
- [M-QALM: A Benchmark to Assess Clinical Reading Comprehension and Knowledge Recall in Large Language Models via Question Answering](https://mlsanity.com/paper/2406.03699/) (2024)
- [ExpNote: Black-box Large Language Models are Better Task Solvers with Experience Notebook](https://mlsanity.com/paper/2311.07032/) (2023)
- [MSDiagnosis: A Benchmark for Evaluating Large Language Models in Multi-Step Clinical Diagnosis](https://mlsanity.com/paper/2408.10039/) (2024)

## Abstract (arXiv)

Medical quality control indicators are essential to assess the qualifications of healthcare institutions for medical services. With the impressive performance of large language models (LLMs) like GPT-4 in the medical field, leveraging these technologies for the Medical Quality Control Indicator Calculation (MQCIC) presents a promising approach. In this work, (1) we introduce a real-world task MQCIC and propose an open-source Chinese electronic medical records (EMRs)-based dataset (CMQCIC-Bench) comprising 785 instances and 76 indicators. (2) We propose a semi-automatic method to enhance the rule representation. Then we propose the Clinical Facts-based Inferential Rule (CF-IR) method that disentangles the clinical fact verification and inferential rule reasoning actions. (3) We conduct comprehensive experiments on 20 representative LLMs, covering general and medical models. Our findings reveal that CF-IR outperforms Chain-of-Thought methods in MQCIC tasks. (4) We conduct an error analysis and investigate the capabilities of clinical fact verification and inferential rule reasoning, providing insights to improve performance in the MQCIC further. The dataset and code is available in this repository https://github.com/YuY-2001/C-MQCIC.