mlsanityData to 14 Oct 2026

EHR-MCP: Real-world Evaluation of Clinical Information Retrieval by Large Language Models via Model Context Protocol

arXiv 2509.15957 ↗First posted 19 Sep 2025cs.AI cs.CL cs.HCcited by 5

Not on the map. Why?

Abstract (arXiv)

Background: Large language models (LLMs) show promise in medicine, but their deployment in hospitals is limited by restricted access to electronic health record (EHR) systems. The Model Context Protocol (MCP) enables integration between LLMs and external tools. Objective: To evaluate whether an LLM connected to an EHR database via MCP can autonomously retrieve clinically relevant information in a real hospital setting. Methods: We developed EHR-MCP, a framework of custom MCP tools integrated with the hospital EHR database, and used GPT-4.1 through a LangGraph ReAct agent to interact with it. Six tasks were tested, derived from use cases of the infection control team (ICT). Eight patients discussed at ICT conferences were retrospectively analyzed. Agreement with physician-generated gold standards was measured. Results: The LLM consistently selected and executed the correct MCP tools. Except for two tasks, all tasks achieved near-perfect accuracy. Performance was lower in the complex task requiring time-dependent calculations. Most errors arose from incorrect arguments or misinterpretation of tool results. Responses from EHR-MCP were reliable, though long and repetitive data risked exceeding the context window. Conclusions: LLMs can retrieve clinical data from an EHR via MCP tools in a real hospital setting, achieving near-perfect performance in simple tasks while highlighting challenges in complex ones. EHR-MCP provides an infrastructure for secure, consistent data access and may serve as a foundation for hospital AI agents. Future work should extend beyond retrieval to reasoning, generation, and clinical impact assessment, paving the way for effective integration of generative AI into clinical practice.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 12 of its 19 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

12 of 19 references.

References · 12

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  5. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 249
  6. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 116
  7. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 93
  8. 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 78
  9. 2025MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agentscited by 62
  10. 2023Human Behavioral Benchmarking: Numeric Magnitude Comparison Effects in Large Language Modelscited by 10
  11. 2024LLM on FHIR -- Demystifying Health Recordscited by 9
  12. 2025Enhancing Clinical Decision Support and EHR Insights through LLMs and the Model Context Protocol: An Open-Source MCP-FHIR Frameworkcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026AgentEHR: Advancing Autonomous Clinical Decision-Making via Retrospective Summarizationcited by 4
  2. 2026MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integrationcited by 3
  3. 2026CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubricscited by 3
  4. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 2
  5. 2026Ascent: An Agentic System over the Model Context Protocol for Real-World Clinical Data Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025EHRStruct: A Comprehensive Benchmark Framework for Evaluating Large Language Models on Structured Electronic Health Record Tasks
  2. 2026Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models
  3. 2025BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text
  4. 2025Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs
  5. 2026Harnessing Large Language Models for Precision Querying and Retrieval-Augmented Knowledge Extraction in Clinical Data Science
  6. 2023Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Study
  7. 2025Reviewing Clinical Knowledge in Medical Large Language Models: Training and Beyond
  8. 2026EHR-RAG: Bridging Long-Horizon Structured Electronic Health Records and Large Language Models via Enhanced Retrieval-Augmented Generation