# Question Answering on Patient Medical Records with Private Fine-Tuned LLMs

arXiv: https://arxiv.org/abs/2501.13687 · HTML: https://mlsanity.com/paper/2501.13687/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2025.

## Built on by (most cited first, 6 of 6)

- [FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering](https://mlsanity.com/paper/2509.19319/) (2025)
- [Scaling Laws for Hypernetwork-Based Knowledge Injection in Large Language Models](https://mlsanity.com/paper/2607.19604/) (2026)
- [Large Language Models for Automating Clinical Data Standardization: HL7 FHIR Use Case](https://mlsanity.com/paper/2507.03067/) (2025)
- [FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records](https://mlsanity.com/paper/2602.23479/) (2026)
- [Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models](https://mlsanity.com/paper/2603.26434/) (2026)
- [Reinforcement Learning for Tool-Calling Agents in Fast Healthcare Interoperability Resources (FHIR)](https://mlsanity.com/paper/2605.14126/) (2026)

## Abstract (arXiv)

Healthcare systems continuously generate vast amounts of electronic health records (EHRs), commonly stored in the Fast Healthcare Interoperability Resources (FHIR) standard. Despite the wealth of information in these records, their complexity and volume make it difficult for users to retrieve and interpret crucial health insights. Recent advances in Large Language Models (LLMs) offer a solution, enabling semantic question answering (QA) over medical data, allowing users to interact with their health records more effectively. However, ensuring privacy and compliance requires edge and private deployments of LLMs. This paper proposes a novel approach to semantic QA over EHRs by first identifying the most relevant FHIR resources for a user query (Task1) and subsequently answering the query based on these resources (Task2). We explore the performance of privately hosted, fine-tuned LLMs, evaluating them against benchmark models such as GPT-4 and GPT-4o. Our results demonstrate that fine-tuned LLMs, while 250x smaller in size, outperform GPT-4 family models by 0.55% in F1 score on Task1 and 42% on Meteor Task in Task2. Additionally, we examine advanced aspects of LLM usage, including sequential fine-tuning, model self-evaluation (narcissistic evaluation), and the impact of training data size on performance. The models and datasets are available here: https://huggingface.co/genloop