# Artificial Intelligence-Based Methods for Fusion of Electronic Health Records and Imaging Data

arXiv: https://arxiv.org/abs/2210.13462 · HTML: https://mlsanity.com/paper/2210.13462/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 82 references that are in the map.
- First posted on arXiv 23 Oct 2022.

## Builds on (most cited first, 4 of 4)

- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners](https://mlsanity.com/paper/1803.10158/) (2018)
- [Machine Learning for Multimodal Electronic Health Records-based Research: Challenges and Perspectives](https://mlsanity.com/paper/2111.04898/) (2021)
- [Prediction of Thrombectomy Functional Outcomes using Multimodal Data](https://mlsanity.com/paper/2005.13061/) (2020)

## Built on by (most cited first, 18 of 18)

- [DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency](https://mlsanity.com/paper/2403.06197/) (2024)
- [The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs](https://mlsanity.com/paper/2303.12961/) (2023)
- [Longitudinal Multimodal Transformer Integrating Imaging and Latent Clinical Signatures From Routine EHRs for Pulmonary Nodule Classification](https://mlsanity.com/paper/2304.02836/) (2023)
- [From CNNs to Transformers in Multimodal Human Action Recognition: A Survey](https://mlsanity.com/paper/2405.15813/) (2024)
- [Structured Semantics from Unstructured Notes: Language Model Approaches to EHR-Based Decision Support](https://mlsanity.com/paper/2506.06340/) (2025)
- [Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation](https://mlsanity.com/paper/2410.17918/) (2024)
- [CLIMD: A Curriculum Learning Framework for Imbalanced Multimodal Diagnosis](https://mlsanity.com/paper/2508.01594/) (2025)
- [Integrating Non-Linear Radon Transformation for Diabetic Retinopathy Grading](https://mlsanity.com/paper/2504.15883/) (2025)
- [DiA-gnostic VLVAE: Disentangled Alignment-Constrained Vision Language Variational AutoEncoder for Robust Radiology Reporting with Missing Modalities](https://mlsanity.com/paper/2511.05968/) (2025)
- [Artificial Intelligence-Based Methods for Precision Medicine: Diabetes Risk Prediction](https://mlsanity.com/paper/2305.16346/) (2023)
- [Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities](https://mlsanity.com/paper/2312.03812/) (2023)
- [Influence of Early through Late Fusion on Pancreas Segmentation from Imperfectly Registered Multimodal MRI](https://mlsanity.com/paper/2409.04563/) (2024)
- [Challenges and proposed solutions in modeling multimodal medical data: A systematic review](https://mlsanity.com/paper/2505.06945/) (2025)
- [TCDiff: Triplex Cascaded Diffusion for High-fidelity Multimodal EHRs Generation with Incomplete Clinical Data](https://mlsanity.com/paper/2508.01615/) (2025)
- [EXR: An Interactive Immersive EHR Visualization in Extended Reality](https://mlsanity.com/paper/2512.05438/) (2025)
- [Rethinking Artificial Intelligence in Medical Imaging: Assumptions, Reality, and Reframing](https://mlsanity.com/paper/2607.27428/) (2026)
- [Hierarchical MoE for Multi-Modal ILD Diagnosis](https://mlsanity.com/paper/2608.25261/) (2026)
- [EviGen: Predictive Evidence Scaffolding for Verifiable Clinical Rationale Generation](https://mlsanity.com/paper/2609.18852/) (2026)

## Abstract (arXiv)

Healthcare data are inherently multimodal, including electronic health records (EHR), medical images, and multi-omics data. Combining these multimodal data sources contributes to a better understanding of human health and provides optimal personalized healthcare. Advances in artificial intelligence (AI) technologies, particularly machine learning (ML), enable the fusion of these different data modalities to provide multimodal insights. To this end, in this scoping review, we focus on synthesizing and analyzing the literature that uses AI techniques to fuse multimodal medical data for different clinical applications. More specifically, we focus on studies that only fused EHR with medical imaging data to develop various AI methods for clinical applications. We present a comprehensive analysis of the various fusion strategies, the diseases and clinical outcomes for which multimodal fusion was used, the ML algorithms used to perform multimodal fusion for each clinical application, and the available multimodal medical datasets. We followed the PRISMA-ScR guidelines. We searched Embase, PubMed, Scopus, and Google Scholar to retrieve relevant studies. We extracted data from 34 studies that fulfilled the inclusion criteria. In our analysis, a typical workflow was observed: feeding raw data, fusing different data modalities by applying conventional machine learning (ML) or deep learning (DL) algorithms, and finally, evaluating the multimodal fusion through clinical outcome predictions. Specifically, early fusion was the most used technique in most applications for multimodal learning (22 out of 34 studies). We found that multimodality fusion models outperformed traditional single-modality models for the same task. Disease diagnosis and prediction were the most common clinical outcomes (reported in 20 and 10 studies, respectively) from a clinical outcome perspective.