# Using Clinical Notes with Time Series Data for ICU Management

arXiv: https://arxiv.org/abs/1909.09702 · HTML: https://mlsanity.com/paper/1909.09702/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2019.

## Built on by (most cited first, 30 of 38)

- [FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion](https://mlsanity.com/paper/2402.03226/) (2024)
- [A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction](https://mlsanity.com/paper/2208.10240/) (2022)
- [Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integration](https://mlsanity.com/paper/2102.04110/) (2021)
- [Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modeling](https://mlsanity.com/paper/2210.12156/) (2022)
- [Multi-modal Time Series Analysis: A Tutorial and Survey](https://mlsanity.com/paper/2503.13709/) (2025)
- [REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models](https://mlsanity.com/paper/2402.07016/) (2024)
- [EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.00036/) (2024)
- [Multimodal data matters: language model pre-training over structured and unstructured electronic health records](https://mlsanity.com/paper/2201.10113/) (2022)
- [On the Importance of Clinical Notes in Multi-modal Learning for EHR Data](https://mlsanity.com/paper/2212.03044/) (2022)
- [FineEHR: Refine Clinical Note Representations to Improve Mortality Prediction](https://mlsanity.com/paper/2304.11794/) (2023)
- [What Do You See in this Patient? Behavioral Testing of Clinical NLP Models](https://mlsanity.com/paper/2111.15512/) (2021)
- [MedLens: Improve Mortality Prediction Via Medical Signs Selecting and Regression](https://mlsanity.com/paper/2305.11742/) (2023)
- [CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis](https://mlsanity.com/paper/2411.00696/) (2024)
- [On the Importance of Step-wise Embeddings for Heterogeneous Clinical Time-Series](https://mlsanity.com/paper/2311.08902/) (2023)
- [Next Visit Diagnosis Prediction via Medical Code-Centric Multimodal Contrastive EHR Modelling with Hierarchical Regularisation](https://mlsanity.com/paper/2401.11648/) (2024)
- [BadCLM: Backdoor Attack in Clinical Language Models for Electronic Health Records](https://mlsanity.com/paper/2407.05213/) (2024)
- [Massively Multimodal Foundation Models: A Framework for Capturing Interactions with Specialized Mixture-of-Experts](https://mlsanity.com/paper/2509.25678/) (2025)
- [Knowledge Graph Representations to enhance Intensive Care Time-Series Predictions](https://mlsanity.com/paper/2311.07180/) (2023)
- [Contrastive Learning on Multimodal Analysis of Electronic Health Records](https://mlsanity.com/paper/2403.14926/) (2024)
- [Benchmarking and Improving LVLMs on Event Extraction from Multimedia Documents](https://mlsanity.com/paper/2509.12876/) (2025)
- [MVMTnet: A Multi-variate Multi-modal Transformer for Multi-class Classification of Cardiac Irregularities Using ECG Waveforms and Clinical Notes](https://mlsanity.com/paper/2302.11021/) (2023)
- [Clinical Note Owns its Hierarchy: Multi-Level Hypergraph Neural Networks for Patient-Level Representation Learning](https://mlsanity.com/paper/2305.09756/) (2023)
- [Multi-modal Graph Learning over UMLS Knowledge Graphs](https://mlsanity.com/paper/2307.04461/) (2023)
- [Multi-Modal Contrastive Learning for Online Clinical Time-Series Applications](https://mlsanity.com/paper/2403.18316/) (2024)
- [MILM: Large Language Models for Multimodal Irregular Time Series with Informative Sampling](https://mlsanity.com/paper/2605.13711/) (2026)
- [Using Clinical Drug Representations for Improving Mortality and Length of Stay Predictions](https://mlsanity.com/paper/2110.08918/) (2021)
- [LETS-C: Leveraging Text Embedding for Time Series Classification](https://mlsanity.com/paper/2407.06533/) (2024)
- [CURENet: Combining Unified Representations for Efficient Chronic Disease Prediction](https://mlsanity.com/paper/2511.11423/) (2025)
- [Transparent Early ICU Mortality Prediction with Clinical Transformer and Per-Case Modality Attribution](https://mlsanity.com/paper/2511.15847/) (2025)
- [Natural Language Processing Methods to Identify Oncology Patients at High Risk for Acute Care with Clinical Notes](https://mlsanity.com/paper/2209.13860/) (2022)

## Abstract (arXiv)

Monitoring patients in ICU is a challenging and high-cost task. Hence, predicting the condition of patients during their ICU stay can help provide better acute care and plan the hospital's resources. There has been continuous progress in machine learning research for ICU management, and most of this work has focused on using time series signals recorded by ICU instruments. In our work, we show that adding clinical notes as another modality improves the performance of the model for three benchmark tasks: in-hospital mortality prediction, modeling decompensation, and length of stay forecasting that play an important role in ICU management. While the time-series data is measured at regular intervals, doctor notes are charted at irregular times, making it challenging to model them together. We propose a method to model them jointly, achieving considerable improvement across benchmark tasks over baseline time-series model. Our implementation can be found at urlhttps://github.com/kaggarwal/ClinicalNotesICU.