A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses
On the map: LLM reasoning → Clinical and biomedical applications
Abstract (arXiv)
We propose a rubric for assessing expressed clinical reasoning in model responses, drawing on three bodies of work: medical education assessment frameworks (ART, SCT, Key Feature Problems and OSCE); clinical LLM benchmarks (MedR-Bench, HealthBench, TIMER-Bench, DR. BENCH, PrIME-LLM and PatientSafeBench); and general LLM reasoning evaluation research, including the Factuality-Validity-Coherence-Utility taxonomy, FaithCoT-Bench and C2-Faith. We use groundedness as a clinically oriented adaptation of the taxonomy's factuality category. The rubric brings these concepts together in a multidimensional framework for scoring free-text responses to gold-standard clinical vignettes. It includes provisional behavioural anchors, applicability rules and a separate flag for case-specific safety-critical errors. General-domain frameworks inform its design but are not treated as validated clinical instruments. The rubric does not replace case-specific reference criteria or the task-specific metrics of existing benchmarks. It has not yet been tested for inter-rater reliability, construct validity or clinical utility. Its immediate purpose is to make evaluation decisions explicit and open to scrutiny before empirical testing.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 16 of its 54 references that are in the map. First posted on arXiv 29 Sep 2026.
Builds on
16 of 54 references.
Closest ancestors · 2
Same direction, newest first
- 2025Evaluation of Causal Reasoning for Large Language Models in Contextualized Clinical Scenarios of Laboratory Test Interpretation19 Sep
- 2025ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room28 May
Other references · 14
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
- 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
- 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 28
- 2025FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoningcited by 26
- 2025EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Modelscited by 9
- 2026C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoningcited by 7
- 2025All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generationscited by 6
- 2025OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queriescited by 5
- 2026Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generationcited by 4
- 2026Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Textcited by 4
- 2026MamaBench: Benchmarking LLM Robustness in Maternal and Child Health Diagnosis through Counterfactual Clinical Perturbationcited by 2
- 2026Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completenesscited by 2
2 more
- 2025H-DDx: A Hierarchical Evaluation Framework for Differential Diagnosiscited by 1
- 2026Evaluating and Improving LLM Self-Modelingcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks
- 2026A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined
- 2025Why Chain of Thought Fails in Clinical Text Understanding
- 2026Medical Reasoning with Large Language Models: A Survey and MR-Bench
- 2025Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoning
- 2025The Knowledge-Reasoning Dissociation: Fundamental Limitations of LLMs in Clinical Natural Language Inference
- 2026CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning
- 2026LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment