# ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs

arXiv: https://arxiv.org/abs/2603.02097 · HTML: https://mlsanity.com/paper/2603.02097/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 92 references that are in the map.
- First posted on arXiv 2 Mar 2026.

## Builds on (most cited first, 15 of 15)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments](https://mlsanity.com/paper/2405.07960/) (2024)
- [A Survey of LLM-based Agents in Medicine: How far are we from Baymax?](https://mlsanity.com/paper/2502.11211/) (2025)
- [Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator](https://mlsanity.com/paper/2403.08495/) (2024)
- [LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation](https://mlsanity.com/paper/2506.04078/) (2025)
- [Polaris: A Safety-focused LLM Constellation Architecture for Healthcare](https://mlsanity.com/paper/2403.13313/) (2024)
- [ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real World](https://mlsanity.com/paper/2406.13890/) (2024)
- [A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domains](https://mlsanity.com/paper/2507.23486/) (2025)
- [MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents](https://mlsanity.com/paper/2511.14439/) (2025)
- [Large Language Models as Agents in the Clinic](https://mlsanity.com/paper/2309.10895/) (2023)
- [Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategies](https://mlsanity.com/paper/2503.07306/) (2025)
- [MLB: A Scenario-Driven Benchmark for Evaluating Large Language Models in Clinical Applications](https://mlsanity.com/paper/2601.06193/) (2026)

## Abstract (arXiv)

Open-ended medical LLM evaluation remains weakly grounded in physician-calibrated coverage of clinically relevant response criteria, especially in localized clinical settings. We introduce textscClinConsensus, a Chinese medical benchmark of 2,500 expert-curated cases spanning 36 specialties, 12 task themes, multiple difficulty levels, and lay-facing versus professional-facing settings. Each case is paired with 30 case-specific binary rubric criteria. To evaluate whether responses satisfy enough physician-authored criteria, we propose Clinician-Anchored Coverage Score (CACS), a physician-calibrated threshold metric instantiated at (k=10), and develop a dual-judge framework combining a GPT-5.1 grader with a physician-supervised Qwen3-8B judge. Evaluating 11 frontier LLMs, we find a persistent coverage gap: Rubric Accuracy ranges from 39.6% to 52.1%, whereas CACS@10 ranges from 17.8% to 32.9%, leaving a 19.2--21.9 point gap across models. Stratified analyses further reveal substantial variation across reasoning, evidence use, structured extraction, medication instructions, follow-up, and dialogue register. These results suggest that medical LLM evaluation should measure thresholded, rubric-grounded clinical coverage rather than average partial correctness.