mlsanityData to 7 Oct 2026

Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review

arXiv 2203.15588 ↗First posted 25 Mar 2022cs.LG cs.AI cs.CVcited by 37

Not on the map. Why?

Abstract (arXiv)

The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet complementary data that are produced during routine practice. For instance, the personalized diagnosis and treatment planning for a single cancer patient relies on the various images (e.g., radiological, pathological, and camera images) and non-image data (e.g., clinical data and genomic data). However, such decision-making procedures can be subjective, qualitative, and have large inter-subject variabilities. With the recent advances in multi-modal deep learning technologies, an increasingly large number of efforts have been devoted to a key question: how do we extract and aggregate multi-modal information to ultimately provide more objective, quantitative computer-aided clinical decision making? This paper reviews the recent studies on dealing with such a question. Briefly, this review will include the (1) overview of current multi-modal learning workflows, (2) summarization of multi-modal fusion methods, (3) discussion of the performance, (4) applications in disease diagnosis and prognosis, and (5) challenges and future directions.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 40 of its 112 references that are in the map. First posted on arXiv 25 Mar 2022.

Builds on

40 of 112 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  4. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  9. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  10. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 738
  11. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 508
  12. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
28 more
  1. 2020Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learningcited by 473
  2. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  3. 2016DeepSurv: Personalized Treatment Recommender System Using A Cox Proportional Hazards Deep Neural Networkcited by 354
  4. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  5. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  6. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  7. 2018Attention Gated Networks: Learning to Leverage Salient Regions in Medical Imagescited by 271
  8. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  9. 2019ScispaCy: Fast and Robust Models for Biomedical Natural Language Processingcited by 244
  10. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  11. 2020Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networkscited by 178
  12. 2019Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosiscited by 165
  13. 2020A Patient-Centric Dataset of Images and Metadata for Identifying Melanomas Using Clinical Contextcited by 146
  14. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  15. 2018Disease Prediction using Graph Convolutional Networks: Application to Autism Spectrum Disorder and Alzheimer's Diseasecited by 98
  16. 2020Deep Learning-based Computational Pathology Predicts Origins for Cancers of Unknown Primarycited by 89
  17. 2021Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyondcited by 75
  18. 2021A Variational Information Bottleneck Approach to Multi-Omics Data Integrationcited by 40
  19. 2021Combining 3D Image and Tabular Data via the Dynamic Affine Feature Map Transformcited by 38
  20. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  21. 2020A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reportscited by 35
  22. 2020Deep Multimodal Neural Architecture Searchcited by 34
  23. 2021Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Datacited by 29
  24. 2021Semi-Supervised Multi-Modal Multi-Instance Multi-Label Deep Network with Optimal Transportcited by 28
  25. 2022Survival Prediction of Brain Cancer with Incomplete Radiology, Pathology, Genomics, and Demographic Datacited by 25
  26. 2022Data Harmonisation for Information Fusion in Digital Healthcare: A State-of-the-Art Systematic Review, Meta-Analysis and Future Research Directionscited by 19
  27. 2022Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformerscited by 8
  28. 2021G-MIND: An End-to-End Multimodal Imaging-Genetics Framework for Biomarker Identification and Disease Classificationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 8; 2025: 12; 2026: 10.

37 papers.

Sort:

By area:

  1. 2025QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Trainingcited by 38
  2. 2024Review of multimodal machine learning approaches in healthcarecited by 30
  3. 2023Multi-modal Machine Learning in Engineering Design: A Review and Future Directionscited by 27
  4. 2023HEALNet: Multimodal Fusion for Heterogeneous Biomedical Datacited by 25
  5. 2023INSPECT: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosiscited by 24
  6. 2022Heterogeneous Graph Learning for Multi-modal Medical Data Analysiscited by 16
  7. 2023Dynamic Multimodal Information Bottleneck for Multimodality Classificationcited by 14
  8. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  9. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  10. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  11. 2025MedOrch: Medical Diagnosis with Tool-Augmented Reasoning Agents for Flexible Extensibilitycited by 9
  12. 2026MultiModalPFN: Extending Prior-Data Fitted Networks for Multimodal Tabular Learningcited by 9
25 more
  1. 2024Structural Attention: Rethinking Transformer for Unpaired Medical Image Synthesiscited by 8
  2. 2023Improved Multimodal Fusion for Small Datasets with Auxiliary Supervisioncited by 6
  3. 2024Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generationcited by 6
  4. 2025CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generationcited by 4
  5. 2026VT-Bench: A Unified Benchmark for Visual-Tabular Multi-Modal Learningcited by 3
  6. 2026MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Imagecited by 2
  7. 2024DSCENet: Dynamic Screening and Clinical-Enhanced Multimodal Fusion for MPNs Subtype Classificationcited by 1
  8. 2025DeFusion: An Effective Decoupling Fusion Network for Multi-Modal Pregnancy Predictioncited by 1
  9. 2025Language-Enhanced Generative Modeling for Amyloid PET Synthesis from MRI and Blood Biomarkerscited by 1
  10. 2026Vision-Assisted Foundation Model for Solving Multi-Task Vehicle Routing Problemscited by 1
  11. 2026MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Modelscited by 1
  12. 2024Sparsely Multimodal Data Fusioncited by 0
  13. 2024Enhancing Multimodal Medical Image Classification using Cross-Graph Modal Contrastive Learningcited by 0
  14. 2024Barttender: An approachable & interpretable way to compare medical imaging and non-imaging datacited by 0
  15. 2025ColonScopeX: Leveraging Explainable Expert Systems with Multimodal Data for Improved Early Diagnosis of Colorectal Cancercited by 0
  16. 2025A Comprehensive Review on RNA Subcellular Localization Predictioncited by 0
  17. 2025Multi-Modal Artificial Intelligence of Embryo Grading and Pregnancy Prediction in Assisted Reproductive Technology: A Reviewcited by 0
  18. 2025Latent Space Data Fusion Outperforms Early Fusion in Multimodal Mental Health Digital Phenotyping Datacited by 0
  19. 2025EQPO: Equitable Group Relative Policy Optimization for Clinical Reasoningcited by 0
  20. 2025MAPI-GNN: Multi-Activation Plane Interaction Graph Neural Network for Multimodal Medical Diagnosiscited by 0
  21. 2026A Multimodal Deep Learning Framework for Edema Classification Using HCT and Clinical Datacited by 0
  22. 2026Quantifying Multimodal Capabilities: Formal Generalization Guarantees in Pairwise Metric Learningcited by 0
  23. 2026Multi-FRuGaL: Multimodal Flexible Redundancy-aware Decomposed Gated Learning for Cancer Diagnosis and Prognosiscited by 0
  24. 2026Multimodal Fusion for Fine-Grained Classification of Breast Fibroadenoma and Phyllodes Tumorscited by 0
  25. 2026Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attentioncited by 0