mlsanityData to 7 Oct 2026

On the Importance of Clinical Notes in Multi-modal Learning for EHR Data

arXiv 2212.03044 ↗First posted 6 Dec 2022cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Understanding deep learning model behavior is critical to accepting machine learning-based decision support systems in the medical community. Previous research has shown that jointly using clinical notes with electronic health record (EHR) data improved predictive performance for patient monitoring in the intensive care unit (ICU). In this work, we explore the underlying reasons for these improvements. While relying on a basic attention-based model to allow for interpretability, we first confirm that performance significantly improves over state-of-the-art EHR data models when combining EHR data and clinical notes. We then provide an analysis showing improvements arise almost exclusively from a subset of notes containing broader context on patient state rather than clinician notes. We believe such findings highlight deep learning models for EHR data to be more limited by partially-descriptive data than by modeling choice, motivating a more data-centric approach in the field.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 6 Dec 2022.

Builds on

26 of 37 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  6. 2018Sanity Checks for Saliency Mapscited by 1,149
  7. 2019Publicly Available Clinical BERT Embeddingscited by 995
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  9. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  10. 2020Quantifying Attention Flow in Transformerscited by 759
  11. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  12. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
14 more
  1. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  2. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  3. 2017Multitask learning and benchmarking with clinical time series datacited by 287
  4. 2019Set Functions for Time Seriescited by 137
  5. 2017Learning to Detect Sepsis with a Multitask Gaussian Process RNN Classifiercited by 47
  6. 2021Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integrationcited by 43
  7. 2019Using Clinical Notes with Time Series Data for ICU Managementcited by 38
  8. 2021HiRID-ICU-Benchmark -- A Comprehensive Machine Learning Benchmark on High-resolution ICU Datacited by 38
  9. 2021Neighborhood Contrastive Learning Applied to Online Patient Monitoringcited by 32
  10. 2021Attention Flows are Shapley Value Explanationscited by 30
  11. 2018Improving Clinical Predictions through Unsupervised Time Series Representation Learningcited by 24
  12. 2018Improving Hospital Mortality Prediction with Medical Named Entities and Multimodal Learningcited by 14
  13. 2016Neural Document Embeddings for Intensive Care Patient Mortality Predictioncited by 7
  14. 2020Real-time Prediction of COVID-19 related Mortality using Electronic Health Recordscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 3; 2025: 0; 2026: 2.

10 papers.

  1. 2023The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRscited by 29
  2. 2023On the Importance of Step-wise Embeddings for Heterogeneous Clinical Time-Seriescited by 5
  3. 2024Next Visit Diagnosis Prediction via Medical Code-Centric Multimodal Contrastive EHR Modelling with Hierarchical Regularisationcited by 5
  4. 2023Knowledge Graph Representations to enhance Intensive Care Time-Series Predictionscited by 3
  5. 2024Temporal Cross-Attention for Dynamic Embedding and Tokenization of Multimodal Electronic Health Recordscited by 3
  6. 2023Multi-modal Graph Learning over UMLS Knowledge Graphscited by 2
  7. 2023This Reads Like That: Deep Learning for Interpretable Natural Language Processingcited by 2
  8. 2024Multi-Modal Contrastive Learning for Online Clinical Time-Series Applicationscited by 2
  9. 2026Temporal Fusion Nexus: A task-agnostic multi-modal embedding model for clinical narratives and irregular time series in post-kidney transplant carecited by 0
  10. 2026Removing Temporal Note Redundancy Improves Multimodal Reinforcement Learning for Medicinecited by 0