# Exploring Scaling Laws for EHR Foundation Models

arXiv: https://arxiv.org/abs/2505.22964 · HTML: https://mlsanity.com/paper/2505.22964/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 22 references that are in the map.
- First posted on arXiv 29 May 2025.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Zero Shot Health Trajectory Prediction Using Transformer](https://mlsanity.com/paper/2407.21124/) (2024)
- [Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events](https://mlsanity.com/paper/2306.11547/) (2023)

## Built on by (most cited first, 12 of 12)

- [Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction](https://mlsanity.com/paper/2603.24562/) (2026)
- [From Statistical Fidelity to Clinical Consistency: Scalable Generation and Auditing of Synthetic Patient Trajectories](https://mlsanity.com/paper/2603.06720/) (2026)
- [TrajOnco: a multi-agent framework for temporal reasoning over longitudinal EHR for multi-cancer early detection](https://mlsanity.com/paper/2604.10386/) (2026)
- [Foundation Models to Unlock Real-World Evidence from Nationwide Medical Claims](https://mlsanity.com/paper/2605.02740/) (2026)
- [Scaling Electronic Health Record Foundation Models for Population Health Management](https://mlsanity.com/paper/2506.00209/) (2025)
- [Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits](https://mlsanity.com/paper/2603.22339/) (2026)
- [On the Invariance and Generality of Neural Scaling Laws](https://mlsanity.com/paper/2605.07546/) (2026)
- [Federated generative event models for tokenized electronic health records](https://mlsanity.com/paper/2608.02939/) (2026)
- [Foresight-England: Development of a National-Scale Generative AI Model of Electronic Health Records for Medical Event Prediction across the COVID-19 Pandemic](https://mlsanity.com/paper/2608.16273/) (2026)
- [Reinforcement Learning over Patient Trajectories for Clinical Reasoning in EHR Foundation Models](https://mlsanity.com/paper/2609.12277/) (2026)
- [Training and Scaling Compute-Optimal Physiological Waveform Foundation Models](https://mlsanity.com/paper/2610.05649/) (2026)
- [Sequence Composition in EHR Foundation Model Pretraining](https://mlsanity.com/paper/2610.16878/) (2026)

## Abstract (arXiv)

The emergence of scaling laws has profoundly shaped the development of large language models (LLMs), enabling predictable performance gains through systematic increases in model size, dataset volume, and compute. Yet, these principles remain largely unexplored in the context of electronic health records (EHRs) -- a rich, sequential, and globally abundant data source that differs structurally from natural language. In this work, we present the first empirical investigation of scaling laws for EHR foundation models. By training transformer architectures on patient timeline data from the MIMIC-IV database across varying model sizes and compute budgets, we identify consistent scaling patterns, including parabolic IsoFLOPs curves and power-law relationships between compute, model parameters, data size, and clinical utility. These findings demonstrate that EHR models exhibit scaling behavior analogous to LLMs, offering predictive insights into resource-efficient training strategies. Our results lay the groundwork for developing powerful EHR foundation models capable of transforming clinical prediction tasks and advancing personalized healthcare.