A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined
On the map: LLM reasoning → Clinical and biomedical applications
Abstract (arXiv)
Exam-style accuracy does not establish whether large language models (LLMs) reason well over clinical records. We define clinical reasoning as integrating and updating evidence across time and sources to form, revise and justify a patient's problem representation and a defensible plan. This structured narrative review maps three literatures: medical education assessment instruments, clinical LLM benchmarks published from 2023 onwards, and general-domain methods for evaluating long-form generation. We examine six dimensions: problem representation, temporal synthesis, differential and management reasoning, counterfactual reasoning, calibrated uncertainty, and reasoning faithfulness. Preprints are included and flagged. No single instrument covers all six dimensions. Problem representation and differential or management reasoning are reasonably covered, although reliability varies by instrument and setting. TIMER-Eval targets temporal synthesis, and ER-Reason assesses sequential diagnostic belief updating. Dedicated uncertainty and counterfactual evaluations are emerging, but their applicability to longitudinal free-text reasoning remains limited. Factual completeness is well theorised in general-domain evaluation, with early clinical evidence of important omissions. Faithfulness remains the weakest dimension, with one identified clinical causal-ablation study on multiple-choice questions. Existing tools should be combined through binary rubric items, separate completeness and correctness scores, case-specific importance weighting with non-compensable safety caps, temporal order-consistency checks, and chance-corrected reliability reporting. Further design work is needed for calibrated uncertainty, counterfactual reasoning and faithfulness over longitudinal free-text records. This review provides a design rationale, not a validated instrument.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 26 of its 45 references that are in the map. First posted on arXiv 1 Oct 2026.
Builds on
26 of 45 references.
Closest ancestors · 5
Same direction, newest first
- 2026A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks2 Jul
- 2026Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning14 Mar
- 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Models10 Jul
- 2025ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room28 May
- 2022DR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processing29 Sep
Other references · 21
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
- 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
- 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
- 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
- 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithfulcited by 166
- 2024Long-form factuality in large language modelscited by 146
- 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 114
- 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 87
- 2025Medical Large Language Model Benchmarks Should Prioritize Construct Validitycited by 36
9 more
- 2025TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Recordscited by 33
- 2026Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Taskscited by 23
- 2025ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklistscited by 22
- 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 19
- 2025All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generationscited by 6
- 2026Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generationcited by 4
- 2026Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Textcited by 4
- 2026MamaBench: Benchmarking LLM Robustness in Maternal and Child Health Diagnosis through Counterfactual Clinical Perturbationcited by 2
- 2026Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completenesscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Medical Reasoning with Large Language Models: A Survey and MR-Bench
- 2026CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning
- 2025Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoning
- 2026A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses
- 2026Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning
- 2026Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning
- 2025Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications
- 2025Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models