mlsanityData to 7 Oct 2026

How to Leverage Multimodal EHR Data for Better Medical Predictions?

arXiv 2110.15763 ↗First posted 29 Oct 2021cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

Healthcare is becoming a more and more important research topic recently. With the growing data in the healthcare domain, it offers a great opportunity for deep learning to improve the quality of medical service. However, the complexity of electronic health records (EHR) data is a challenge for the application of deep learning. Specifically, the data produced in the hospital admissions are monitored by the EHR system, which includes structured data like daily body temperature, and unstructured data like free text and laboratory measurements. Although there are some preprocessing frameworks proposed for specific EHR data, the clinical notes that contain significant clinical value are beyond the realm of their consideration. Besides, whether these different data from various views are all beneficial to the medical tasks and how to best utilize these data remain unclear. Therefore, in this paper, we first extract the accompanying clinical notes from EHR and propose a method to integrate these data, we also comprehensively study the different models and the data leverage methods for better medical task prediction. The results on two medical prediction tasks show that our fused model with different data outperforms the state-of-the-art method that without clinical notes, which illustrates the importance of our fusion method and the value of clinical note features. Our code is available at https: //github.com/emnlp-mimic/mimic.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 29 Oct 2021.

Builds on

17 of 31 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  5. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  6. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  7. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  8. 2020DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscited by 326
  9. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  10. 2015Doctor AI: Predicting Clinical Events via Recurrent Neural Networkscited by 205
  11. 2019MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-IIIcited by 121
  12. 2019Star-Transformercited by 94
5 more
  1. 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
  2. 2018GAMENet: Graph Augmented MEmory Networks for Recommending Medication Combinationcited by 79
  3. 2018RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Datacited by 47
  4. 2021MUFASA: Multimodal Fusion Architecture Search for Electronic Health Recordscited by 30
  5. 2019Multimodal Ensemble Approach to Incorporate Various Types of Clinical Notes for Predicting Readmissioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 1; 2024: 9; 2025: 10; 2026: 3.

26 papers.

Sort:

By area:

  1. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  2. 2022A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Predictioncited by 48
  3. 2025TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loopcited by 39
  4. 2022Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modelingcited by 33
  5. 2025Multi-modal Time Series Analysis: A Tutorial and Surveycited by 24
  6. 2024REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Modelscited by 21
  7. 2025How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlookcited by 20
  8. 2024EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generationcited by 16
  9. 2022Multimodal data matters: language model pre-training over structured and unstructured electronic health recordscited by 13
  10. 2023Hierarchical Pretraining on Multimodal Electronic Health Recordscited by 11
  11. 2024Recent Advances in Predictive Modeling with Electronic Health Recordscited by 11
  12. 2024Next Visit Diagnosis Prediction via Medical Code-Centric Multimodal Contrastive EHR Modelling with Hierarchical Regularisationcited by 5
14 more
  1. 2024BadCLM: Backdoor Attack in Clinical Language Models for Electronic Health Recordscited by 5
  2. 2024Global Contrastive Training for Multimodal Electronic Health Records with Language Supervisioncited by 4
  3. 2025Massively Multimodal Foundation Models: A Framework for Capturing Interactions with Specialized Mixture-of-Expertscited by 4
  4. 2024Automated Fusion of Multimodal Electronic Health Records for Better Medical Predictionscited by 3
  5. 2024Overcoming Uncertain Incompleteness for Robust Multimodal Sequential Diagnosis Prediction via Curriculum Data Erasing Guided Knowledge Distillationcited by 3
  6. 2025Rethinking Gating Mechanism in Sparse MoE: Handling Arbitrary Modality Inputs with Confidence-Guided Gatecited by 3
  7. 2025MEDMKG: Benchmarking Medical Knowledge Exploitation with Multimodal Knowledge Graphcited by 2
  8. 2025Temporal Entailment Pretraining for Clinical Language Models over EHR Datacited by 1
  9. 2025CURENet: Combining Unified Representations for Efficient Chronic Disease Predictioncited by 1
  10. 2025One Size Fits None: Rethinking Fairness in Medical AIcited by 0
  11. 2025Annotation-Assisted Learning of Treatment Policies From Multimodal Electronic Health Recordscited by 0
  12. 2026CLIC: Contextual Language-Informed Cardiac Pathology Classificationcited by 0
  13. 2026CRS-Triage: Confidence- and Reliability-Aware Selective Triage under Incomplete Clinical Evidencecited by 0
  14. 2026Backdoor Learning in Language Models and Vision-Language Modelscited by 0