# PULSE-ICU: A Pretrained Unified Long-Sequence Encoder for Multi-task Prediction in Intensive Care Units

arXiv: https://arxiv.org/abs/2511.22199 · HTML: https://mlsanity.com/paper/2511.22199/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 27 Nov 2025.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Time2Vec: Learning a Vector Representation of Time](https://mlsanity.com/paper/1907.05321/) (2019)
- [Zero Shot Health Trajectory Prediction Using Transformer](https://mlsanity.com/paper/2407.21124/) (2024)
- [Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences](https://mlsanity.com/paper/2201.11838/) (2022)
- [Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records](https://mlsanity.com/paper/2106.11360/) (2021)
- [MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images](https://mlsanity.com/paper/2207.07027/) (2022)
- [GenHPF: General Healthcare Predictive Framework with Multi-task Multi-source Learning](https://mlsanity.com/paper/2207.09858/) (2022)
- [EHRMamba: Towards Generalizable and Scalable Foundation Models for Electronic Health Records](https://mlsanity.com/paper/2405.14567/) (2024)
- [Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML](https://mlsanity.com/paper/2306.05109/) (2023)
- [Multimodal Pretraining of Medical Time Series and Notes](https://mlsanity.com/paper/2312.06855/) (2023)
- [CEHR-GPT: Generating Electronic Health Records with Chronological Patient Timelines](https://mlsanity.com/paper/2402.04400/) (2024)
- [DuETT: Dual Event Time Transformer for Electronic Health Records](https://mlsanity.com/paper/2304.13017/) (2023)
- [ExBEHRT: Extended Transformer for Electronic Health Records to Predict Disease Subtypes & Progressions](https://mlsanity.com/paper/2303.12364/) (2023)
- [CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis](https://mlsanity.com/paper/2411.00696/) (2024)
- [A Novel Multi-Task Teacher-Student Architecture with Self-Supervised Pretraining for 48-Hour Vasoactive-Inotropic Trend Analysis in Sepsis Mortality Prediction](https://mlsanity.com/paper/2502.16834/) (2025)
- [An Efficient Contrastive Unimodal Pretraining Method for EHR Time Series Data](https://mlsanity.com/paper/2410.09199/) (2024)
- [Mind the Missing: Variable-Aware Representation Learning for Irregular EHR Time Series using Large Language Models](https://mlsanity.com/paper/2509.22121/) (2025)
- [Improving Representation Learning of Complex Critical Care Data with ICU-BERT](https://mlsanity.com/paper/2502.19593/) (2025)

## Abstract (arXiv)

Intensive care unit (ICU) data are highly irregular, heterogeneous, and temporally fragmented, posing challenges for generalizable clinical prediction. We present PULSE-ICU, a self-supervised foundation model that learns event-level ICU representations from large-scale EHR sequences without resampling or manual feature engineering. A unified embedding module encodes event identity, continuous values, units, and temporal attributes, while a Longformer-based encoder enables efficient modeling of long trajectories. PULSE-ICU was fine-tuned across 18 prediction tasks, including mortality, intervention forecasting, and phenotype identification, achieving strong performance across task types. External validation on eICU, HiRID, and P12 showed substantial improvements with minimal fine-tuning, demonstrating robustness to domain shift and variable constraints. These findings suggest that foundation-style modeling can improve data efficiency and adaptability, providing a scalable framework for ICU decision support across diverse clinical environments.