# Integrating Physiological Time Series and Clinical Notes with Deep Learning for Improved ICU Mortality Prediction

arXiv: https://arxiv.org/abs/2003.11059 · HTML: https://mlsanity.com/paper/2003.11059/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Mar 2020.

## Built on by (most cited first, 6 of 6)

- [Multimodal Pretraining of Medical Time Series and Notes](https://mlsanity.com/paper/2312.06855/) (2023)
- [MORE-CLEAR: Multimodal Offline Reinforcement learning for Clinical notes Leveraged Enhanced State Representation](https://mlsanity.com/paper/2508.07681/) (2025)
- [OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Prediction](https://mlsanity.com/paper/2604.16878/) (2026)
- [ArterialNet: Reconstructing Arterial Blood Pressure Waveform with Wearable Pulsatile Signals, a Cohort-Aware Approach](https://mlsanity.com/paper/2410.18895/) (2024)
- [XAI for In-hospital Mortality Prediction via Multimodal ICU Data](https://mlsanity.com/paper/2312.17624/) (2023)
- [MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction](https://mlsanity.com/paper/2605.20197/) (2026)

## Abstract (arXiv)

Intensive Care Unit Electronic Health Records (ICU EHRs) store multimodal data about patients including clinical notes, sparse and irregularly sampled physiological time series, lab results, and more. To date, most methods designed to learn predictive models from ICU EHR data have focused on a single modality. In this paper, we leverage the recently proposed interpolation-prediction deep learning architecture(Shukla and Marlin 2019) as a basis for exploring how physiological time series data and clinical notes can be integrated into a unified mortality prediction model. We study both early and late fusion approaches and demonstrate how the relative predictive value of clinical text and physiological data change over time. Our results show that a late fusion approach can provide a statistically significant improvement in mortality prediction performance over using individual modalities in isolation.