# FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering

arXiv: https://arxiv.org/abs/2509.19319 · HTML: https://mlsanity.com/paper/2509.19319/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 6 of its 25 references that are in the map.
- First posted on arXiv 12 Sep 2025.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 6 of 6)

- [Large language models in healthcare and medical domain: A review](https://mlsanity.com/paper/2401.06775/) (2023)
- [Text-to-SQL Generation for Question Answering on Electronic Medical Records](https://mlsanity.com/paper/1908.01839/) (2019)
- [MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents](https://mlsanity.com/paper/2501.14654/) (2025)
- [EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records](https://mlsanity.com/paper/2301.07695/) (2023)
- [Overview of the EHRSQL 2024 Shared Task on Reliable Text-to-SQL Modeling on Electronic Health Records](https://mlsanity.com/paper/2405.06673/) (2024)
- [Question Answering on Patient Medical Records with Private Fine-Tuned LLMs](https://mlsanity.com/paper/2501.13687/) (2025)

## Built on by (most cited first, 22 of 22)

- [PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments](https://mlsanity.com/paper/2605.02240/) (2026)
- [HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks](https://mlsanity.com/paper/2604.09937/) (2026)
- [GRASP: Gated Regression-Aware Skill Proposer for Self-Improving LLM Agents](https://mlsanity.com/paper/2605.29668/) (2026)
- [EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning](https://mlsanity.com/paper/2606.23301/) (2026)
- [ClinSeekAgent: Automating Multimodal Evidence Seeking for Agentic Clinical Reasoning](https://mlsanity.com/paper/2605.20176/) (2026)
- [AutoMedBench: Towards Medical AutoResearch with Agentic AI Models](https://mlsanity.com/paper/2606.01961/) (2026)
- [The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy](https://mlsanity.com/paper/2607.11175/) (2026)
- [GDPevo: Evaluating Agent Self-Evolution on Real Business Tasks](https://mlsanity.com/paper/2608.03764/) (2026)
- [Empowering Locally Deployable Medical Agent via State Enhanced Logical Skills for FHIR-based Clinical Tasks](https://mlsanity.com/paper/2603.06902/) (2026)
- [Agentic clinical reasoning over longitudinal myeloma records: a retrospective evaluation against expert consensus](https://mlsanity.com/paper/2604.24473/) (2026)
- [MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings](https://mlsanity.com/paper/2605.30295/) (2026)
- [Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applications](https://mlsanity.com/paper/2510.17764/) (2025)
- [FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records](https://mlsanity.com/paper/2602.23479/) (2026)
- [Detecting Clinical Discrepancies in Health Coaching Agents: A Dual-Stream Memory and Reconciliation Architecture](https://mlsanity.com/paper/2604.27045/) (2026)
- [Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance](https://mlsanity.com/paper/2606.18613/) (2026)
- [World Feedback for Clinical Agents: Diagnosing RL in FHIR Environments](https://mlsanity.com/paper/2607.01470/) (2026)
- [MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agents](https://mlsanity.com/paper/2607.18999/) (2026)
- [PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents](https://mlsanity.com/paper/2607.25485/) (2026)
- [BeTaL-GBI: Admission-Aware Benchmark Tuning and Full-Stack Verification of Geometric Belief Interfaces](https://mlsanity.com/paper/2608.21503/) (2026)
- [Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark](https://mlsanity.com/paper/2609.30027/) (2026)
- [Phantom Merge: When Your Large Language Model Agents Pick One but Tell You About Another](https://mlsanity.com/paper/2610.16141/) (2026)
- [ProCARE: Real-World Study Automation via Profile-Grounded Evidence Contracts](https://mlsanity.com/paper/2610.16175/) (2026)

## Abstract (arXiv)

The recent shift toward the Health Level Seven Fast Healthcare Interoperability Resources (HL7 FHIR) standard opens a new frontier for clinical AI, demanding LLM agents to navigate complex, resource-based data models instead of conventional structured health data. However, existing benchmarks have lagged behind this transition, lacking the realism needed to evaluate recent LLMs on interoperable clinical data. To bridge this gap, we introduce FHIR-AgentBench, a benchmark that grounds 2,931 real-world clinical questions in the HL7 FHIR standard. Using this benchmark, we systematically evaluate agentic frameworks, comparing different data retrieval strategies (direct FHIR API calls vs. specialized tools), interaction patterns (single-turn vs. multi-turn), and reasoning strategies (natural language vs. code generation). Our experiments highlight the practical challenges of retrieving data from intricate FHIR resources and the difficulty of reasoning over them, both of which critically affect question answering performance. We publicly release the FHIR-AgentBench dataset and evaluation suite (https://github.com/glee4810/FHIR-AgentBench) to promote reproducible research and the development of robust, reliable LLM agents for clinical applications.