mlsanityData to 7 Oct 2026

ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs

arXiv 2603.02097 ↗First posted 2 Mar 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Open-ended medical LLM evaluation remains weakly grounded in physician-calibrated coverage of clinically relevant response criteria, especially in localized clinical settings. We introduce textscClinConsensus, a Chinese medical benchmark of 2,500 expert-curated cases spanning 36 specialties, 12 task themes, multiple difficulty levels, and lay-facing versus professional-facing settings. Each case is paired with 30 case-specific binary rubric criteria. To evaluate whether responses satisfy enough physician-authored criteria, we propose Clinician-Anchored Coverage Score (CACS), a physician-calibrated threshold metric instantiated at (k=10), and develop a dual-judge framework combining a GPT-5.1 grader with a physician-supervised Qwen3-8B judge. Evaluating 11 frontier LLMs, we find a persistent coverage gap: Rubric Accuracy ranges from 39.6% to 52.1%, whereas CACS@10 ranges from 17.8% to 32.9%, leaving a 19.2--21.9 point gap across models. Stratified analyses further reveal substantial variation across reasoning, evidence use, structured extraction, medication instructions, follow-up, and dialogue register. These results suggest that medical LLM evaluation should measure thresholded, rubric-grounded clinical coverage rather than average partial correctness.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 92 references that are in the map. First posted on arXiv 2 Mar 2026.

Builds on

15 of 92 references.

References · 15

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  3. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  4. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  5. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  6. 2025A Survey of LLM-based Agents in Medicine: How far are we from Baymax?cited by 84
  7. 2024Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulatorcited by 36
  8. 2025LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validationcited by 29
  9. 2024ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real Worldcited by 28
  10. 2025A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domainscited by 28
  11. 2024Polaris: A Safety-focused LLM Constellation Architecture for Healthcarecited by 27
  12. 2025MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agentscited by 8
3 more
  1. 2023Large Language Models as Agents in the Cliniccited by 4
  2. 2025Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategiescited by 2
  3. 2026MLB: A Scenario-Driven Benchmark for Evaluating Large Language Models in Clinical Applicationscited by 1

Built on by

Not cited yet.