mlsanityData to 14 Oct 2026

Clinical Reasoning Graphs: Structured Evaluation of LLM Diagnostic Reasoning Reveals Competence Without Consistency

arXiv 2606.29876 ↗First posted 29 Jun 2026cs.CL cs.AI q-bio.QMcited by 0

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Modern large language models (LLMs) reach 60-70% diagnostic accuracy on complex clinical case benchmarks, but accuracy alone cannot distinguish stable clinically-grounded reasoning from pattern matching. We introduce clinical reasoning graphs, structured graph representations extracted from free-text LLM diagnostic traces using a domain-grounded ontology with 5 node types and 7 edge types. We apply this pipeline to 750 traces from five LLMs across 50 New England Journal of Medicine Clinicopathological Conference cases and three prompt conditions, and test whether diagnostic traces show stable structured reasoning patterns, or diagnostic schemas, for clinically similar cases. We operationalize this as higher graph similarity among clinically similar cases than among clinically dissimilar ones. Across 15 model-condition comparisons, within-cluster and between-cluster composite similarity are nearly equal, and no comparison survives multiple-testing correction; a component-level analysis finds any residual content signal far below schema scale. Graph similarity is also nearly identical for pairs of models that are both correct (0.488) and both incorrect (0.484), suggesting that graph structure captures a dimension not reflected in diagnostic accuracy. Structured reflection prompting increases explicit discriminating-feature analysis within traces (+33%) but does not increase cross-case consistency. These results show diagnostic competence without schema-scale reasoning consistency, and indicate that final-answer accuracy should be complemented by process-level evaluation. We release the ontology, extraction pipeline, validation protocol, and the extracted reasoning graphs and similarity artifacts as resources for structured evaluation of LLM clinical reasoning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 3 of its 12 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

3 of 12 references.

References · 3

  1. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  2. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  3. 2026Measuring and curing reasoning rigidity: from decorative chain-of-thought to genuine faithfulnesscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning
  2. 2023Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Study
  3. 2025MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports
  4. 2026VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphs
  5. 2025Brittleness and Promise: Knowledge Graph Based Reward Modeling for Diagnostic Reasoning
  6. 2026CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation
  7. 2026SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence
  8. 2026Clinical Knowledge Graph Construction and Evaluation with Multi-LLMs via Retrieval-Augmented Generation