A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks
On the map: LLM reasoning → Clinical and biomedical applications
Abstract (arXiv)
Multiple-choice medical benchmarks are increasingly saturated, and recent rubric-based evaluations such as HealthBench have shown that open-ended clinical performance is far from solved - its "Hard" subset top score remains 32%. We present a small, deliberately difficult evaluation dataset of five clinician-authored clinical scenarios spanning four specialties (anaesthesia, internal/family medicine, emergency medicine, and obstetrics), each accompanied by an atomic, weighted, MECE rubric (25-62 criteria per task; 184 criteria total) authored from a clinician-drafted golden answer. We evaluate three frontier models: GPT 5.4, Claude Opus 4.7, and Gemini 3.1 Pro. Mean rubric pass rates were 0.47 (Claude), 0.38 (GPT), and 0.37 (Gemini). The central finding is an inversion of clinical priority: the highest-weighted (weight-5, critical) criteria passed at only 32.4-41.7%, while low-stakes weight-1 criteria passed at 80-90%. 55 of 108 critical (weight-5) criteria (51%) were satisfied by no model. Three LLM autoraters reproduced expert met/not-met labels on 92.8-94.6% of 552 graded criteria. We position this as a methods-and-preliminary-findings contribution: the five tasks demonstrate a scalable, defensible pipeline ready to develop into a large-scale benchmark.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 9 of its 16 references that are in the map. First posted on arXiv 2 Jul 2026.
Builds on
9 of 16 references.
Closest ancestors · 2
Same direction, newest first
- 2025ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room28 May
- 2025Advancing Conversational Diagnostic AI with Multimodal Reasoning6 May
Other references · 7
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
- 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
- 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
- 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
- 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
- 2025Towards Conversational AI for Disease Managementcited by 11
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses
- 2026Retrieval-Augmented Agentic Rubric Generation for Reliable Medical Response Evaluation
- 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks
- 2026Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewards
- 2025Why Chain of Thought Fails in Clinical Text Understanding
- 2026Medical Reasoning with Large Language Models: A Survey and MR-Bench
- 2024Superhuman performance of a large language model on the reasoning tasks of a physician
- 2024Guiding Clinical Reasoning with Large Language Models via Knowledge Seeds