mlsanityData to 7 Oct 2026

Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attention

arXiv 2305.02504 ↗First posted 4 May 2023cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Electronic Health Record (EHR) provides abundant information through various modalities. However, learning multi-modal EHR is currently facing two major challenges, namely, 1) data embedding and 2) cases with missing modality. A lack of shared embedding function across modalities can discard the temporal relationship between different EHR modalities. On the other hand, most EHR studies are limited to relying only on EHR Times-series, and therefore, missing modality in EHR has not been well-explored. Therefore, in this study, we introduce a Unified Multi-modal Set Embedding (UMSE) and Modality-Aware Attention (MAA) with Skip Bottleneck (SB). UMSE treats all EHR modalities without a separate imputation module or error-prone carry-forward, whereas MAA with SB learns missing modal EHR with effective modality-aware attention. Our model outperforms other baseline models in mortality, vasopressor need, and intubation need prediction with the MIMIC-IV dataset.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

24 of 35 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  4. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  6. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  7. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  8. 2016Recurrent Neural Networks for Multivariate Time Series with Missing Valuescited by 548
  9. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  10. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  11. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  12. 2020Self-Supervised MultiModal Versatile Networkscited by 276
12 more
  1. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
  2. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  3. 2019Set Functions for Time Seriescited by 137
  4. 2021Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Seriescited by 96
  5. 2019Interpolation-Prediction Networks for Irregularly Sampled Time Seriescited by 89
  6. 2022Integrated multimodal artificial intelligence framework for healthcare applicationscited by 69
  7. 2022MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray imagescited by 53
  8. 2022A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Predictioncited by 48
  9. 2022Geometric Multimodal Contrastive Representation Learningcited by 40
  10. 2017Clinical Intervention Prediction and Understanding using Deep Networkscited by 35
  11. 2020MHVAE: a Human-Inspired Deep Hierarchical Generative Model for Multimodal Representation Learningcited by 5
  12. 2021Improving Early Sepsis Prediction with Multi Modal Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 8; 2026: 9.

25 papers.

Sort:

By area:

  1. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
  2. 2025A Closer Look at Multimodal Representation Collapsecited by 25
  3. 2024Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modalitycited by 21
  4. 2024REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Modelscited by 21
  5. 2025How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlookcited by 20
  6. 2024EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generationcited by 16
  7. 2023Event-Based Contrastive Learning for Medical Time Seriescited by 13
  8. 2024FlexCare: Leveraging Cross-Task Synergy for Flexible Multimodal Healthcare Predictioncited by 6
  9. 2024Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generationcited by 6
  10. 2026The Taxonomies, Training, and Applications of Event Stream Modelling for Electronic Health Recordscited by 3
  11. 2026AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Taskscited by 3
  12. 2025Diffmv: A Unified Diffusion Framework for Healthcare Predictions with Random Missing Views and View Lazinesscited by 2
13 more
  1. 2025MedPatch: Confidence-Guided Multi-Stage Fusion for Multimodal Clinical Datacited by 2
  2. 2024From Basic to Extra Features: Hypergraph Transformer Pretrain-then-Finetuning for Balanced Clinical Predictions on EHRcited by 1
  3. 2025ICYM2I: The illusion of multimodal informativeness under missingnesscited by 1
  4. 2025Multimodal Disease Progression Modeling via Spatiotemporal Disentanglement and Multiscale Alignmentcited by 1
  5. 2026Head-wise Modality Specialization within MLLMs for Robust Fake News Detection under Missing Modalitycited by 1
  6. 2026Handling and Interpreting Missing Modalities in Patient Clinical Trajectories via Autoregressive Sequence Modelingcited by 1
  7. 2026Multimodal Routing for Interpretable, Robust, and Auditable Clinical Predictioncited by 1
  8. 2025Missing-Modality-Aware Graph Neural Network for Cancer Classificationcited by 0
  9. 2025EMR-AGENT: Automating Cohort and Feature Extraction from EMR Databasescited by 0
  10. 2026When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusioncited by 0
  11. 2026Structure-Aware Set Transformers: Temporal and Variable-Type Attention Biases for Asynchronous Clinical Time Seriescited by 0
  12. 2026A Clinical Point Cloud Paradigm for In-Hospital Mortality Prediction from Multi-Level Incomplete Multimodal EHRscited by 0
  13. 2026CRS-Triage: Confidence- and Reliability-Aware Selective Triage under Incomplete Clinical Evidencecited by 0