The Doctor Will (Still) See You Now: On the Structural Limits of Agentic AI in Healthcare
On the map: LLM agents and tool use → Medical and scientific domain agents
Abstract (arXiv)
Across healthcare, agentic artificial intelligence (AI) systems are increasingly promoted as capable of autonomous action, yet in practice they currently operate under near-total human oversight due to safety, regulatory, and liability constraints that make autonomous clinical reasoning infeasible in high-stakes environments. While market enthusiasm suggests a revolution in healthcare agents, the conceptual assumptions and accountability structures shaping these systems remain underexamined. We present a qualitative study based on interviews with 20 stakeholders, including developers, implementers, and end users. Our analysis identifies three mutually reinforcing tensions: conceptual fragmentation regarding the definition of `agentic'; an autonomy contradiction where commercial promises exceed operational reality; and an evaluation blind spot that prioritizes technical benchmarks over sociotechnical safety. We argue that agentic {AI} functions as a site of contested meaning-making where technical aspirations, commercial incentives, and clinical constraints intersect, carrying material consequences for patient safety and the distribution of blame.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 9 of its 53 references that are in the map. First posted on arXiv 6 Feb 2026.
Builds on
9 of 53 references.
References · 9
- 2022Large Language Models Encode Clinical Knowledgecited by 1,807
- 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
- 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
- 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
- 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
- 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Taskscited by 60
- 2024More than Marketing? On the Information Value of AI Benchmarks for Practitionerscited by 16
- 2025Artificial Intelligence Should Genuinely Support Clinical Reasoning and Decision Making To Bridge the Translational Gapcited by 13
- 2025Accountability Framework for Healthcare AI Systems: Towards Joint Accountability in Decision Makingcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Agentic AI, Medical Morality, and the Transformation of the Patient-Physician Relationship ↗
- 2025Socio-technical aspects of Agentic AI
- 2026Agentic AI in Healthcare & Medicine: A Seven-Dimensional Taxonomy for Empirical Evaluation of LLM-based Agents
- 2026Agentic AI Governance and Lifecycle Management in Healthcare
- 2026From Single Chatbots to Governed Agent Ecosystems: An Agentic AI Pattern Catalogue and Orchestration Framework for Mission-Critical Hospital Information Management Systems
- 2026Agentic AI in medicine: architectures, applications, evaluation, and challenges for clinical translation
- 2026Agentic AI Enhances Physician Trust in Clinical Decision Making
- 2026Grounding Clinical AI Competency in Human Cognition Through the Clinical World Model and Skill-Mix Framework