# CEHR-XGPT: A Scalable Multi-Task Foundation Model for Electronic Health Records

arXiv: https://arxiv.org/abs/2509.03643 · HTML: https://mlsanity.com/paper/2509.03643/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 3 Sep 2025.

## Builds on (most cited first, 23 of 23)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Generating Multi-label Discrete Patient Records using Generative Adversarial Networks](https://mlsanity.com/paper/1703.06490/) (2017)
- [Transformer Language Models without Positional Encodings Still Learn Positional Information](https://mlsanity.com/paper/2203.16634/) (2022)
- [Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Data](https://mlsanity.com/paper/2001.05295/) (2020)
- [MOTOR: A Time-To-Event Foundation Model For Structured Medical Records](https://mlsanity.com/paper/2301.03150/) (2023)
- [Zero Shot Health Trajectory Prediction Using Transformer](https://mlsanity.com/paper/2407.21124/) (2024)
- [CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks](https://mlsanity.com/paper/2111.08585/) (2021)
- [Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records](https://mlsanity.com/paper/2106.11360/) (2021)
- [Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events](https://mlsanity.com/paper/2306.11547/) (2023)
- [A Multifaceted Benchmarking of Synthetic Electronic Health Record Generation Models](https://mlsanity.com/paper/2208.01230/) (2022)
- [Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs](https://mlsanity.com/paper/2412.16178/) (2024)
- [CorGAN: Correlation-Capturing Convolutional Generative Adversarial Networks for Generating Synthetic Healthcare Records](https://mlsanity.com/paper/2001.09346/) (2020)
- [Analyzing the Robustness of Nearest Neighbors to Adversarial Examples](https://mlsanity.com/paper/1706.03922/) (2017)
- [GenHPF: General Healthcare Predictive Framework with Multi-task Multi-source Learning](https://mlsanity.com/paper/2207.09858/) (2022)
- [EHRMamba: Towards Generalizable and Scalable Foundation Models for Electronic Health Records](https://mlsanity.com/paper/2405.14567/) (2024)
- [CEHR-GPT: Generating Electronic Health Records with Chronological Patient Timelines](https://mlsanity.com/paper/2402.04400/) (2024)
- [Generating Electronic Health Records with Multiple Data Types and Constraints](https://mlsanity.com/paper/2003.07904/) (2020)
- [CORE-BEHRT: A Carefully Optimized and Rigorously Evaluated BEHRT](https://mlsanity.com/paper/2404.15201/) (2024)
- [Time-Dependent Representation for Neural Event Sequence Prediction](https://mlsanity.com/paper/1708.00065/) (2017)
- [Foresight -- Generative Pretrained Transformer (GPT) for Modelling of Patient Timelines using EHRs](https://mlsanity.com/paper/2212.08072/) (2022)
- [CONAN: Complementary Pattern Augmentation for Rare Disease Detection](https://mlsanity.com/paper/1911.13232/) (2019)
- [Targeted-BEHRT: Deep learning for observational causal inference on longitudinal electronic health records](https://mlsanity.com/paper/2202.03487/) (2022)

## Built on by (most cited first, 4 of 4)

- [Retrieval-aligned Tabular Foundation Models Enable Robust Clinical Risk Prediction in Electronic Health Records Under Real-world Constraints](https://mlsanity.com/paper/2604.01841/) (2026)
- [Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction](https://mlsanity.com/paper/2603.24562/) (2026)
- [PORTER: Language-Grounded Event Representations for Portable Structured EHR Foundation Models](https://mlsanity.com/paper/2606.24102/) (2026)
- [Toward Scalable Early Cancer Detection: Evaluating EHR-Based Predictive Models Against Traditional Screening Criteria](https://mlsanity.com/paper/2511.11293/) (2025)

## Abstract (arXiv)

Electronic Health Records (EHRs) provide a rich, longitudinal view of patient health and hold significant potential for advancing clinical decision support, risk prediction, and data-driven healthcare research. However, most artificial intelligence (AI) models for EHRs are designed for narrow, single-purpose tasks, limiting their generalizability and utility in real-world settings. Here, we present CEHR-XGPT, a general-purpose foundation model for EHR data that unifies three essential capabilities - feature representation, zero-shot prediction, and synthetic data generation - within a single architecture. To support temporal reasoning over clinical sequences, CEHR-XGPT incorporates a novel time-token-based learning framework that explicitly encodes patients' dynamic timelines into the model structure. CEHR-XGPT demonstrates strong performance across all three tasks and generalizes effectively to external datasets through vocabulary expansion and fine-tuning. Its versatility enables rapid model development, cohort discovery, and patient outcome forecasting without the need for task-specific retraining.