# FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records

arXiv: https://arxiv.org/abs/2602.23479 · HTML: https://mlsanity.com/paper/2602.23479/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 29 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Builds on (most cited first, 7 of 7)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Text-to-SQL Generation for Question Answering on Electronic Medical Records](https://mlsanity.com/paper/1908.01839/) (2019)
- [EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records](https://mlsanity.com/paper/2301.07695/) (2023)
- [FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering](https://mlsanity.com/paper/2509.19319/) (2025)
- [A Dataset for Addressing Patient's Information Needs related to Clinical Course of Hospitalization](https://mlsanity.com/paper/2506.04156/) (2025)
- [Question Answering for Electronic Health Records: A Scoping Review of datasets and models](https://mlsanity.com/paper/2310.08759/) (2023)
- [Question Answering on Patient Medical Records with Private Fine-Tuned LLMs](https://mlsanity.com/paper/2501.13687/) (2025)

## Abstract (arXiv)

Though patients are increasingly granted digital access to their electronic health records (EHRs), existing interfaces may not support precise, trustworthy answers to patient-specific questions. Large language models (LLM) show promise in clinical question answering (QA), but retrieval-based approaches are computationally inefficient, prone to hallucination, and difficult to deploy over real-life EHRs. This work introduces FHIRPath-QA, the first open dataset and benchmark for patient-specific QA that includes open-standard FHIRPath queries over real-world clinical data. A text-to-FHIRPath QA paradigm is proposed that shifts reasoning from free-text generation to FHIRPath query synthesis. For o4-mini, this reduced average token usage by 391x relative to retrieval-first prompting (629,829 vs 1,609 tokens per question) and lowered failure rates from 0.36 to 0.09 on clinician-phrased questions. Built on MIMIC-IV on FHIR Demo, the dataset pairs over 14k natural language questions in patient and clinician phrasing with validated FHIRPath queries and answers. Empirically, the evaluated LLMs achieve at most 42% accuracy, highlighting the challenge of the task, but benefit strongly from supervised fine-tuning, with query synthesis accuracy improving from 27% to 79% for 4o-mini. These results highlight that text-to-FHIRPath synthesis has the potential to serve as a practical foundation for safe, efficient, and interoperable consumer health applications, and the FHIRPath-QA dataset and benchmark serve as a starting point for future research on the topic. The full dataset and generation code can be accessed at: https://github.com/mooshifrew/fhirpath-qa.