mlsanityData to 13 Oct 2026

Machine Learning for Multimodal Electronic Health Records-based Research: Challenges and Perspectives

arXiv 2111.04898 ↗First posted 9 Nov 2021cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Background: Electronic Health Records (EHRs) contain rich information of patients' health history, which usually include both structured and unstructured data. There have been many studies focusing on distilling valuable information from structured data, such as disease codes, laboratory test results, and treatments. However, relying on structured data only might be insufficient in reflecting patients' comprehensive information and such data may occasionally contain erroneous records. Objective: With the recent advances of machine learning (ML) and deep learning (DL) techniques, an increasing number of studies seek to obtain more accurate results by incorporating unstructured free-text data as well. This paper reviews studies that use multimodal data, i.e. a combination of structured and unstructured data, from EHRs as input for conventional ML or DL models to address the targeted tasks. Materials and Methods: We searched in the Institute of Electrical and Electronics Engineers (IEEE) Digital Library, PubMed, and Association for Computing Machinery (ACM) Digital Library for articles related to ML-based multimodal EHR studies. Results and Discussion: With the final 94 included studies, we focus on how data from different modalities were combined and interacted using conventional ML and DL techniques, and how these algorithms were applied in EHR-related tasks. Further, we investigate the advantages and limitations of these fusion methods and indicate future directions for ML-based multimodal EHR research.

Key facts. Cited by 7 papers in the map (data through 13 Oct 2026). Builds on 30 of its 93 references that are in the map. First posted on arXiv 9 Nov 2021.

Builds on

30 of 93 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,539
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,360
  5. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  6. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  7. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  8. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 490
  9. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 428
  10. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 369
  11. 2018Scalable and accurate deep learning for electronic health recordscited by 278
  12. 2017Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysiscited by 146
18 more
  1. 2019Pre-training of Graph Augmented Transformers for Medication Recommendationcited by 124
  2. 2017Comparative Study of CNN and RNN for Natural Language Processingcited by 81
  3. 2018Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMscited by 67
  4. 2020Deep Representation Learning of Patient Data from Electronic Health Records (EHR): A Systematic Reviewcited by 46
  5. 2018Multimodal Machine Learning for Automated ICD Codingcited by 41
  6. 2020Deep Representation Learning of Electronic Health Records to Unlock Patient Stratification at Scalecited by 39
  7. 2019Natural Language Processing of Clinical Notes on Chronic Diseases: Systematic Reviewcited by 36
  8. 2020A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reportscited by 35
  9. 2019TAPER: Time-Aware Patient EHR Representationcited by 22
  10. 2018Natural Language Processing for EHR-Based Computational Phenotypingcited by 18
  11. 2018Deep EHR: Chronic Disease Prediction Using Medical Notescited by 15
  12. 2020Improving Clinical Outcome Predictions Using Convolution over Medical Entities with Multimodal Learningcited by 11
  13. 2019Identifying Sub-Phenotypes of Acute Kidney Injury using Structured and Unstructured Electronic Health Record Data with Memory Networkscited by 10
  14. 2020Adversarial Infidelity Learning for Model Interpretationcited by 9
  15. 2020Multimodal Learning for Cardiovascular Risk Prediction using EHR Datacited by 4
  16. 2019Multimodal Ensemble Approach to Incorporate Various Types of Clinical Notes for Predicting Readmissioncited by 3
  17. 2017Multimodal Visual Concept Learning with Weakly Supervised Techniquescited by 2
  18. 2019Enhancing Prediction Models for One-Year Mortality in Patients with Acute Myocardial Infarction and Post Myocardial Infarction Syndromecited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 3; 2023: 1; 2024: 1; 2025: 1; 2026: 1.

7 papers.

  1. 2022Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modelingcited by 33
  2. 2022Artificial Intelligence-Based Methods for Fusion of Electronic Health Records and Imaging Datacited by 18
  3. 2023MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videoscited by 12
  4. 2024Multimodal Sleep Apnea Detection with Missing or Noisy Modalitiescited by 2
  5. 2022Patients' Severity States Classification based on Electronic Health Record (EHR) Data using Multiple Machine Learning and Deep Learning Approachescited by 0
  6. 2025Improving ARDS Diagnosis Through Context-Aware Concept Bottleneck Modelscited by 0
  7. 2026Multimodal Training to Unimodal Deployment: Leveraging Unstructured Data During Training to Optimize Structured Data Only Deploymentcited by 0