# Reasoning in Real World Clinical Care: Why Large Language Models Are Not Yet Safe for Autonomous Clinical Decision Support

arXiv: https://arxiv.org/abs/2607.28677 · HTML: https://mlsanity.com/paper/2607.28677/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 8 of its 29 references that are in the map.
- First posted on arXiv 29 Jul 2026.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 8 of 8)

- [MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning](https://mlsanity.com/paper/2406.00922/) (2024)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [LLMs Don't Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanations](https://mlsanity.com/paper/2509.09396/) (2025)
- [AI-based Clinical Decision Support for Primary Care: A Real-World Study](https://mlsanity.com/paper/2507.16947/) (2025)
- [Towards Conversational AI for Disease Management](https://mlsanity.com/paper/2503.06074/) (2025)
- [A prospective clinical feasibility study of a conversational diagnostic AI in an ambulatory primary care clinic](https://mlsanity.com/paper/2603.08448/) (2026)
- [SymptomAI: Toward a Conversational AI Agent for Everyday Symptom Assessment](https://mlsanity.com/paper/2605.04012/) (2026)
- [Measuring Epistemic Resilience of LLMs Under Misleading Medical Context](https://mlsanity.com/paper/2606.12291/) (2026)

## Abstract (arXiv)

LLM now pass medical licensing examinations and, in curated cases, can rival physicians at diagnostic reasoning. These developments have accelerated the use of LLMs for symptom assessment and clinical decision support in diagnostic and treatment guidance, administrative documentation, and rules-based alert enhancement. This Perspective concerns the most consequential of these applications: the autonomous triage of self-presenting, undifferentiated patients, with little or no clinician in the loop. For that task, the evidence of safety does not yet exist. The gap is not in medical knowledge but in the fidelity of clinical evaluation: a model optimized to continue the most probable text is not optimized to act safely when the safe answer is the improbable must-not-miss diagnosis. Safe triage is not the selection of the most likely diagnosis; it is a sequential decision under asymmetric cost, in which the single catastrophic miss outweighs many false alarms, and the decisive signal may be one the patient has not volunteered - and that the model has not been trained to seek. The core deficit is therefore one of information gathering under uncertainty. Under incomplete histories, LLM systems may fail to show the behaviors safe triage requires: broadening the differential; seeking the missing red flag; lowering the threshold for escalation; deferring judgement until sufficient information is obtained; and escalating concern where high-harm diagnoses remain unexcluded. These modes of failure for LLMs can be difficult to detect considering that evaluations to date often use complete, well-curated, confidence-gated simulations. The application of LLMs under these conditions may be amplified by assistant-like behaviors and positive bias, including credulity, agreeableness, and miscalibration - when these are not constrained by clinical triage logic.