# A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined

arXiv: https://arxiv.org/abs/2610.01938 · HTML: https://mlsanity.com/paper/2610.01938/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 26 of its 45 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 26 of 26)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [Chain-of-Thought Reasoning In The Wild Is Not Always Faithful](https://mlsanity.com/paper/2503.08679/) (2025)
- [Long-form factuality in large language models](https://mlsanity.com/paper/2403.18802/) (2024)
- [VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation](https://mlsanity.com/paper/2406.19276/) (2024)
- [CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists](https://mlsanity.com/paper/2403.18771/) (2024)
- [Medical Large Language Model Benchmarks Should Prioritize Construct Validity](https://mlsanity.com/paper/2503.10694/) (2025)
- [TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Records](https://mlsanity.com/paper/2503.04176/) (2025)
- [Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias](https://mlsanity.com/paper/2606.19544/) (2026)
- [Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks](https://mlsanity.com/paper/2603.00077/) (2026)
- [ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists](https://mlsanity.com/paper/2506.01241/) (2025)
- [ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room](https://mlsanity.com/paper/2505.22919/) (2025)
- [Automating Expert-Level Medical Reasoning Evaluation of Large Language Models](https://mlsanity.com/paper/2507.07988/) (2025)
- [DR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processing](https://mlsanity.com/paper/2209.14901/) (2022)
- [All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations](https://mlsanity.com/paper/2510.07083/) (2025)
- [Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generation](https://mlsanity.com/paper/2604.03141/) (2026)
- [Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text](https://mlsanity.com/paper/2606.18471/) (2026)
- [Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning](https://mlsanity.com/paper/2603.13988/) (2026)
- [MamaBench: Benchmarking LLM Robustness in Maternal and Child Health Diagnosis through Counterfactual Clinical Perturbation](https://mlsanity.com/paper/2607.14385/) (2026)
- [Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness](https://mlsanity.com/paper/2607.19322/) (2026)
- [A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks](https://mlsanity.com/paper/2607.02175/) (2026)

## Abstract (arXiv)

Exam-style accuracy does not establish whether large language models (LLMs) reason well over clinical records. We define clinical reasoning as integrating and updating evidence across time and sources to form, revise and justify a patient's problem representation and a defensible plan. This structured narrative review maps three literatures: medical education assessment instruments, clinical LLM benchmarks published from 2023 onwards, and general-domain methods for evaluating long-form generation. We examine six dimensions: problem representation, temporal synthesis, differential and management reasoning, counterfactual reasoning, calibrated uncertainty, and reasoning faithfulness. Preprints are included and flagged. No single instrument covers all six dimensions. Problem representation and differential or management reasoning are reasonably covered, although reliability varies by instrument and setting. TIMER-Eval targets temporal synthesis, and ER-Reason assesses sequential diagnostic belief updating. Dedicated uncertainty and counterfactual evaluations are emerging, but their applicability to longitudinal free-text reasoning remains limited. Factual completeness is well theorised in general-domain evaluation, with early clinical evidence of important omissions. Faithfulness remains the weakest dimension, with one identified clinical causal-ablation study on multiple-choice questions. Existing tools should be combined through binary rubric items, separate completeness and correctness scores, case-specific importance weighting with non-compensable safety caps, temporal order-consistency checks, and chance-corrected reliability reporting. Further design work is needed for calibrated uncertainty, counterfactual reasoning and faithfulness over longitudinal free-text records. This review provides a design rationale, not a validated instrument.