# EHRMamba: Towards Generalizable and Scalable Foundation Models for Electronic Health Records

arXiv: https://arxiv.org/abs/2405.14567 · HTML: https://mlsanity.com/paper/2405.14567/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 54 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Time2Vec: Learning a Vector Representation of Time](https://mlsanity.com/paper/1907.05321/) (2019)
- [Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling](https://mlsanity.com/paper/2403.03234/) (2024)
- [CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks](https://mlsanity.com/paper/2111.08585/) (2021)
- [Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records](https://mlsanity.com/paper/2106.11360/) (2021)
- [Clinical Concept Embeddings Learned from Massive Sources of Multimodal Medical Data](https://mlsanity.com/paper/1804.01486/) (2018)
- [SST: Multi-Scale Hybrid Mamba-Transformer Experts for Time Series Forecasting](https://mlsanity.com/paper/2404.14757/) (2024)
- [Multi-task Prediction of Disease Onsets from Longitudinal Lab Tests](https://mlsanity.com/paper/1608.00647/) (2016)
- [Semantic processing of EHR data for clinical research](https://mlsanity.com/paper/1511.03036/) (2015)

## Built on by (most cited first, 30 of 32)

- [Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions](https://mlsanity.com/paper/2507.00191/) (2025)
- [Foundation models for electronic health records: representation dynamics and transferability](https://mlsanity.com/paper/2504.10422/) (2025)
- [OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning](https://mlsanity.com/paper/2511.23269/) (2025)
- [Towards a Medical AI Scientist](https://mlsanity.com/paper/2603.28589/) (2026)
- [FoMoH: A clinically meaningful foundation model evaluation for structured electronic health records](https://mlsanity.com/paper/2505.16941/) (2025)
- [An Investigation of Memorization Risk in Healthcare Foundation Models](https://mlsanity.com/paper/2510.12950/) (2025)
- [One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Models](https://mlsanity.com/paper/2602.00541/) (2026)
- [CEHR-XGPT: A Scalable Multi-Task Foundation Model for Electronic Health Records](https://mlsanity.com/paper/2509.03643/) (2025)
- [Retrieval-aligned Tabular Foundation Models Enable Robust Clinical Risk Prediction in Electronic Health Records Under Real-world Constraints](https://mlsanity.com/paper/2604.01841/) (2026)
- [EHR2Path: Comprehensive Pathway-Level Modeling of Longitudinal Patient Trajectories from Multimodal Electronic Health Records](https://mlsanity.com/paper/2506.04831/) (2025)
- [Foundation Models for Clinical Records at Health System Scale](https://mlsanity.com/paper/2507.00574/) (2025)
- [Quantifying surprise in clinical care: Detecting highly informative events in electronic health records with foundation models](https://mlsanity.com/paper/2507.22798/) (2025)
- [Efficient Generative Prediction for EHR Foundation Models: The SCOPE and REACH Estimators](https://mlsanity.com/paper/2602.03730/) (2026)
- [Learning Clinical Representations Under Systematic Distribution Shift](https://mlsanity.com/paper/2603.07348/) (2026)
- [Discriminative Representation Learning for Clinical Prediction](https://mlsanity.com/paper/2603.20921/) (2026)
- [Clin-JEPA: A Multi-Phase Co-Training Framework for Joint-Embedding Predictive Pretraining on EHR Patient Trajectories](https://mlsanity.com/paper/2605.10840/) (2026)
- [Explainable AI for Infection Prevention and Control: Modeling CPE Acquisition and Patient Outcomes in an Irish Hospital with Transformers](https://mlsanity.com/paper/2509.14942/) (2025)
- [Uncertainty-Aware Foundation Models for Clinical Data](https://mlsanity.com/paper/2604.04175/) (2026)
- [General Demographic Pre-trained Models for Enhancing Predictive Performance Across Diseases and Population](https://mlsanity.com/paper/2509.07330/) (2025)
- [Towards Self-Supervised Foundation Models for Critical Care Time Series](https://mlsanity.com/paper/2509.19885/) (2025)
- [Trajectory-guided discharge stratification for heart failure using short-context electronic health record sequence modeling](https://mlsanity.com/paper/2511.16839/) (2025)
- [PULSE-ICU: A Pretrained Unified Long-Sequence Encoder for Multi-task Prediction in Intensive Care Units](https://mlsanity.com/paper/2511.22199/) (2025)
- [Coefficient of Variation Masking: A Volatility-Aware Strategy for EHR Foundation Models](https://mlsanity.com/paper/2512.05216/) (2025)
- [Dense Feature Learning via Linear Structure Preservation in Medical Data](https://mlsanity.com/paper/2602.07706/) (2026)
- [Representation Before Training: A Practical Benchmark for Generative Medical Event Model Tokenization](https://mlsanity.com/paper/2604.16775/) (2026)
- [AGA3DNet: Anatomy-Guided Gaussian Priors with Multi-view xLSTM for 3D Brain MRI Subtype Classification](https://mlsanity.com/paper/2605.07142/) (2026)
- [Event Fields: Learning Latent Event Structure for Waveform Foundation Models](https://mlsanity.com/paper/2605.08685/) (2026)
- [WISTERIA: Learning Clinical Representations from Noisy Supervision via Multi-View Consistency in Electronic Health Records](https://mlsanity.com/paper/2605.09765/) (2026)
- [EHR-RAGp: Retrieval-Augmented Prototype-Guided Foundation Model for Electronic Health Records](https://mlsanity.com/paper/2605.12335/) (2026)
- [AURORA: Contextual Orthogonalization for Geometric Representation Learning in Healthcare Foundation Models](https://mlsanity.com/paper/2605.17765/) (2026)

## Abstract (arXiv)

Transformers have significantly advanced the modeling of Electronic Health Records (EHR), yet their deployment in real-world healthcare is limited by several key challenges. Firstly, the quadratic computational cost and insufficient context length of these models hinder hospitals' ability in processing the extensive medical histories typical in EHR data. Additionally, existing models employ separate finetuning for each clinical task, complicating maintenance in healthcare environments. Moreover, these models focus exclusively on either clinical prediction or EHR forecasting, lacking proficiency in both tasks. To overcome these limitations, we introduce EHRMamba, a robust foundation model built on the Mamba architecture. EHRMamba can process sequences up to 300% longer than previous models due to its linear computational cost. We also introduce a novel approach to Multitask Prompted Finetuning (MPF) for EHR data, which enables EHRMamba to simultaneously learn multiple clinical tasks in a single finetuning phase, significantly enhancing deployment and cross-task generalization. Furthermore, our model leverages the HL7 FHIR data standard to simplify integration into existing hospital systems. Alongside EHRMamba, we open-source Odyssey, a toolkit designed to support the development and deployment of EHR foundation models, with an emphasis on data standardization and interpretability. Our evaluations on the MIMIC-IV dataset demonstrate that EHRMamba advances state-of-the-art performance across 6 major clinical tasks and excels in EHR forecasting, marking a significant leap forward in the field.