Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models
On the map: LLM reasoning → Clinical and biomedical applications
Abstract (arXiv)
Large language models (LLMs) show significant potential in healthcare, prompting numerous benchmarks to evaluate their capabilities. However, concerns persist regarding the reliability of these benchmarks, which often lack clinical fidelity, robust data management, and safety-oriented evaluation metrics. To address these shortcomings, we introduce MedCheck, the first lifecycle-oriented assessment framework specifically designed for medical benchmarks. Our framework deconstructs a benchmark's development into five continuous stages, from design to governance, and provides a comprehensive checklist of 46 medically-tailored criteria. Using MedCheck, we conducted an in-depth empirical evaluation of 53 medical LLM benchmarks. Our analysis uncovers widespread, systemic issues, including a profound disconnect from clinical practice, a crisis of data integrity due to unmitigated contamination risks, and a systematic neglect of safety-critical evaluation dimensions like model robustness and uncertainty awareness. Based on these findings, MedCheck serves as both a diagnostic tool for existing benchmarks and an actionable guideline to foster a more standardized, reliable, and transparent approach to evaluating AI in healthcare.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 9 of its 87 references that are in the map. First posted on arXiv 6 Aug 2025.
Builds on
9 of 87 references.
References · 9
- 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
- 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
- 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Taskscited by 64
- 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
- 2024MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applicationscited by 25
- 2024Large Language Model Benchmarks in Medical Taskscited by 17
- 2022Knowledge Distillation of Russian Language Models with Reduction of Vocabularycited by 7
- 2025RiskAgent: Synergizing Language Models with Validated Tools for Evidence-Based Risk Predictioncited by 3
- 2025FOSS solution for Molecular Dynamics Simulation Automation and Collaboration with MDSGATcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026A primer on evaluation methods for large language models in healthcare
- 2025BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text
- 2025Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategies
- 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Models
- 2024Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine
- 2025Addressing Benchmarking Gaps in Large Language Models for Health and Medicine with Dynamic Red-Teaming
- 2025MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents
- 2026Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks