mlsanityData to 7 Oct 2026

Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects

arXiv 2311.02332 ↗First posted 4 Nov 2023cs.LG cs.CVcited by 15

Not on the map. Why?

Abstract (arXiv)

Machine learning (ML) applications in medical artificial intelligence (AI) systems have shifted from traditional and statistical methods to increasing application of deep learning models. This survey navigates the current landscape of multimodal ML, focusing on its profound impact on medical image analysis and clinical decision support systems. Emphasizing challenges and innovations in addressing multimodal representation, fusion, translation, alignment, and co-learning, the paper explores the transformative potential of multimodal models for clinical predictions. It also highlights the need for principled assessments and practical implementation of such models, bringing attention to the dynamics between decision support systems and healthcare providers and personnel. Despite advancements, challenges such as data biases and the scarcity of "big data" in many biomedical domains persist. We conclude with a discussion on principled innovation and collaborative efforts to further the mission of seamless integration of multimodal ML models into biomedical practice.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 48 of its 104 references that are in the map. First posted on arXiv 4 Nov 2023.

Builds on

48 of 104 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  9. 2018Image Transformercited by 770
  10. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 738
  11. 2016Generative Adversarial Text to Image Synthesiscited by 735
  12. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
36 more
  1. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  2. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  3. 2021TransBTS: Multimodal Brain Tumor Segmentation Using Transformercited by 265
  4. 2017Generating Multi-label Discrete Patient Records using Generative Adversarial Networkscited by 197
  5. 2019Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosiscited by 165
  6. 2022mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentationcited by 94
  7. 2021Can Language Models be Biomedical Knowledge Bases?cited by 83
  8. 2022The Creativity of Text-to-Image Generationcited by 73
  9. 2021Knowledge distillation from multi-modal to mono-modal segmentation networkscited by 68
  10. 2023Brain decoding: toward real-time reconstruction of visual perceptioncited by 64
  11. 2021Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processingcited by 57
  12. 2021Text to Image Generation with Semantic-Spatial Aware GANcited by 53
  13. 2023Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modalitycited by 44
  14. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  15. 2020Test-time Unsupervised Domain Adaptationcited by 37
  16. 2018Deep Learning under Privileged Information Using Heteroscedastic Dropoutcited by 26
  17. 2022Survival Prediction of Brain Cancer with Incomplete Radiology, Pathology, Genomics, and Demographic Datacited by 25
  18. 2021Improving Pneumonia Localization via Cross-Attention on Medical Images and Reportscited by 21
  19. 2020PFA-GAN: Progressive Face Aging with Generative Adversarial Networkcited by 19
  20. 2020Unified cross-modality feature disentangler for unsupervised multi-domain MRI abdomen organs segmentationcited by 16
  21. 2021Modality Completion via Gaussian Process Prior Variational Autoencoders for Multi-Modal Glioma Segmentationcited by 16
  22. 2022SFusion: Self-attention based N-to-One Multimodal Fusion Blockcited by 14
  23. 2020GANDALF: Generative Adversarial Networks with Discriminator-Adaptive Loss Fine-tuning for Alzheimer's Disease Diagnosis from MRIcited by 13
  24. 2020Cross-Domain Medical Image Translation by Shared Latent Gaussian Mixture Modelcited by 12
  25. 2018CariGANs: Unpaired Photo-to-Caricature Translationcited by 11
  26. 2020Harnessing Uncertainty in Domain Adaptation for MRI Prostate Lesion Segmentationcited by 9
  27. 2020Interpretation of Brain Morphology in Association to Alzheimer's Disease Dementia Classification Using Graph Convolutional Networks on Triangulated Meshescited by 8
  28. 2023Longitudinal Multimodal Transformer Integrating Imaging and Latent Clinical Signatures From Routine EHRs for Pulmonary Nodule Classificationcited by 8
  29. 2020Lesion Mask-based Simultaneous Synthesis of Anatomic and MolecularMR Images using a GANcited by 6
  30. 2020M2Net: Multi-modal Multi-channel Network for Overall Survival Time Prediction of Brain Tumor Patientscited by 4
  31. 2020Region-of-interest guided Supervoxel Inpainting for Self-supervisioncited by 4
  32. 2020Flow-based Deformation Guidance for Unpaired Multi-Contrast MRI Image-to-Image Translationcited by 4
  33. 2020Peri-Diagnostic Decision Support Through Cost-Efficient Feature Acquisition at Test-Timecited by 3
  34. 2020Soft Tissue Sarcoma Co-Segmentation in Combined MRI and PET/CT Datacited by 3
  35. 2020MIA-Prognosis: A Deep Learning Framework to Predict Therapy Responsecited by 3
  36. 2020Automatic Breast Lesion Classification by Joint Neural Analysis of Mammography and Ultrasoundcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 5; 2026: 6.

15 papers.

Sort:

By area:

  1. 2023GeoLocator: a location-integrated large multimodal model for inferring geo-privacycited by 13
  2. 2025UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysiscited by 12
  3. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  4. 2024Electrocardiogram-Language Model for Few-Shot Question Answering with Meta Learningcited by 4
  5. 2026An Empirical Analysis of Calibration and Selective Prediction in Multimodal Clinical Condition Classificationcited by 2
  6. 2025Multimodal Disease Progression Modeling via Spatiotemporal Disentanglement and Multiscale Alignmentcited by 1
  7. 2026Early Risk Stratification of Dosing Errors in Clinical Trials Using Machine Learningcited by 1
  8. 2024Leveraging Computational Pathology AI for Noninvasive Optical Imaging Analysis Without Retrainingcited by 0
  9. 2025Modality-AGnostic Image Cascade (MAGIC) for Multi-Modality Cardiac Substructure Segmentationcited by 0
  10. 2025Evaluating Strategies for Synthesizing Clinical Notes for Medical Multimodal AIcited by 0
  11. 2025Dual Model Deep Learning for Alzheimer Prognosticationcited by 0
  12. 2026MuteBench: Modality Unavailability Tolerance Evaluation for Incomplete Multimodal Fusioncited by 0
3 more
  1. 2026Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Modelscited by 0
  2. 2026Did the Grid Erase the Event? EndoClock for Auditing Medical World-Model Pipelinescited by 0
  3. 2026BioEVAL: A global, multi-institutional benchmark of large language and multimodal models for bioengineeringcited by 0