CardioBench: A Real-World Data Benchmark for Evaluating Large Language Models in Clinically Authentic Cardiovascular Care Scenarios
Not on the map. Why?
Abstract (arXiv)
Background: Most medical large language model (LLM) benchmarks focus on examination knowledge or isolated tasks and may not reflect the longitudinal, multimodal, and safety-critical workflow of cardiovascular care. Objective: To develop CardioBench, a real-world benchmark spanning the cardiovascular care continuum, and assess LLM performance across clinical dimensions and specialist tasks. Methods: CardioBench includes 2,263 items from 13 task-specific datasets derived from de-identified cardiovascular records and examination data. Sixteen cardiology physicians conducted annotation and reference construction, followed by cross-review from two senior cardiologists. Seven LLMs generated 15,841 outputs under standardized zero-shot settings. Open-ended tasks were evaluated using key-point coverage and holistic clinical quality, while CardioEthics was scored by accuracy. Results: GPT-5.4 achieved the highest macro-average (62.55) and item-weighted mean (62.19), followed by Gemini 3.1 Pro (59.95) and Qwen 3.6 27B (59.72). GPT-5.4 ranked first in all three dimensions. CardioAuxReport performed best (86.38), whereas CardioECGRead (17.25) and CardioEthics (17.34) were lowest. The largest gaps between holistic clinical quality and key-point coverage occurred in CardioComm (52.71), CardioEmergRescue (52.05), and CardioTreatPlan (48.80). Conclusions: To our knowledge, CardioBench is the largest real-world, multi-task benchmark for LLM evaluation across the cardiovascular care continuum and offers the broadest coverage of clinically authentic cardiology scenarios reported to date. It provides a rigorous framework for identifying model strengths, clinically important omissions, and priorities for future development.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 11 of its 42 references that are in the map. First posted on arXiv 28 Jul 2026.
Builds on
11 of 42 references.
References · 11
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
- 2022Large Language Models Encode Clinical Knowledgecited by 1,817
- 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
- 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 116
- 2024A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Reviewcited by 66
- 2023Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AIcited by 32
- 2024Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicinecited by 31
- 2025A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domainscited by 28
- 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Modelscited by 22
- 2024ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Taskscited by 8
- 2025Evaluating Clinical Competencies of Large Language Models with a General Practice Benchmarkcited by 3
Built on by
Not cited yet.