# A Living Benchmark for Information Retrieval from Electronic Health Records

arXiv: https://arxiv.org/abs/2609.30205 · HTML: https://mlsanity.com/paper/2609.30205/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 78 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 19 of 19)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [Kimi K2.5: Visual Agentic Intelligence](https://mlsanity.com/paper/2602.02276/) (2026)
- [EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries](https://mlsanity.com/paper/2402.16040/) (2024)
- [R2MED: A Benchmark for Reasoning-Driven Medical Retrieval](https://mlsanity.com/paper/2505.14558/) (2025)
- [Agentic Information Retrieval](https://mlsanity.com/paper/2410.09713/) (2024)
- [Building the EHR Foundation Model via Next Event Prediction](https://mlsanity.com/paper/2509.25591/) (2025)
- [OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries](https://mlsanity.com/paper/2509.02594/) (2025)
- [Automating Benchmark Design](https://mlsanity.com/paper/2510.25039/) (2025)
- [Adoption and Use of LLMs at an Academic Medical Center](https://mlsanity.com/paper/2602.00074/) (2026)
- [On the Robustness of Answer Formats in Medical Reasoning Models](https://mlsanity.com/paper/2509.20866/) (2025)
- [Clinical Note Bloat Reduction for Efficient LLM Use](https://mlsanity.com/paper/2604.16364/) (2026)
- [AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows](https://mlsanity.com/paper/2606.17474/) (2026)
- [Vital Trace: Protocol-Constrained Patient-State Reasoning for Longitudinal Clinical Trajectories](https://mlsanity.com/paper/2602.12833/) (2026)
- [Agentic clinical reasoning over longitudinal myeloma records: a retrospective evaluation against expert consensus](https://mlsanity.com/paper/2604.24473/) (2026)
- [A Randomized Controlled Trial and Pilot of Scout: an LLM-Based EHR Search and Synthesis Platform](https://mlsanity.com/paper/2604.26953/) (2026)
- [Configurable Clinical Information Extraction with Agentic RAG: What Works, What Breaks, and Why](https://mlsanity.com/paper/2606.19602/) (2026)

## Abstract (arXiv)

Large language model (LLM)-based clinical assistants are increasingly being integrated into electronic health record (EHR) systems, transforming how clinicians retrieve and synthesize information from patient records. Their safety and utility depend on rigorous evaluation, yet existing benchmarks are manually curated, costly to update, and rapidly become obsolete with evolving technological advancements. We present a scalable framework that automatically generates question--answer pairs from longitudinal EHR notes. Nineteen clinicians validate the benchmark generator, producing the Benchmark for Retrieving Information in EHRs (BRIE), a continuously maintainable evaluation dataset. Across nine LLMs and five inference strategies, state-of-the-art systems frequently omit clinically important information, particularly for questions requiring synthesis across multiple documents and encounters. Because the generator itself is validated, BRIE supports evaluations that static benchmarks cannot, including the generation of multiple answers that reflect variation in clinician reasoning for robust performance assessment and continuously refreshing benchmark content to guard against leakage. Our results demonstrate that scalable benchmark generation enables rigorous, up-to-date evaluation of clinical LLMs as they are deployed in rapidly evolving healthcare settings.