# Pretraining EHR Foundation Models with Patient-Aware Sampling

arXiv: https://arxiv.org/abs/2607.22114 · HTML: https://mlsanity.com/paper/2607.22114/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 7 references that are in the map.
- First posted on arXiv 24 Jul 2026.

## Builds on (most cited first, 4 of 4)

- [In-context Pretraining: Language Modeling Beyond Document Boundaries](https://mlsanity.com/paper/2310.10638/) (2023)
- [Zero Shot Health Trajectory Prediction Using Transformer](https://mlsanity.com/paper/2407.21124/) (2024)
- [Analysing The Impact of Sequence Composition on Language Model Pre-Training](https://mlsanity.com/paper/2402.13991/) (2024)
- [Foundation Model of Electronic Medical Records for Adaptive Risk Estimation](https://mlsanity.com/paper/2502.06124/) (2025)

## Abstract (arXiv)

Autoregressive foundation models for electronic health records (EHRs) typically inherit pretraining methods from language modeling, where patient trajectories are concatenated into a single token stream and windows are sampled from that stream. In EHR data, this choice is consequential: windows may mix multiple patients, and patients with longer records contribute more optimization updates, potentially introducing bias. We propose Patient Sampling, a pretraining sequence-construction method that allows us to control how training signal is distributed across patients. We compare this method to the standard approach, which we refer to as Global Stream. We show that stochastic Patient Sampling with controllable weighting improves performance on real-world EHR data. Across downstream clinical tasks on MIMIC-IV v2.2 and v3.1, Patient Sampling improves Macro AUROC and AUPRC over the Global Stream baseline. These results identify training and validation sequence construction as important and underexplored design choices for autoregressive EHR foundation models.