mlsanityData to 7 Oct 2026

A scoping review on multimodal deep learning in biomedical images and texts

arXiv 2307.07362 ↗First posted 14 Jul 2023cs.CV cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Computer-assisted diagnostic and prognostic systems of the future should be capable of simultaneously processing multimodal data. Multimodal deep learning (MDL), which involves the integration of multiple sources of data, such as images and text, has the potential to revolutionize the analysis and interpretation of biomedical data. However, it only caught researchers' attention recently. To this end, there is a critical need to conduct a systematic review on this topic, identify the limitations of current work, and explore future directions. In this scoping review, we aim to provide a comprehensive overview of the current state of the field and identify key concepts, types of studies, and research gaps with a focus on biomedical images and texts joint learning, mainly because these two were the most commonly available data types in MDL research. This study reviewed the current uses of multimodal deep learning on five tasks: (1) Report generation, (2) Visual question answering, (3) Cross-modal retrieval, (4) Computer-aided diagnosis, and (5) Semantic segmentation. Our results highlight the diverse applications and potential of MDL and suggest directions for future research in the field. We hope our review will facilitate the collaboration of natural language processing (NLP) and medical imaging communities and support the next generation of decision-making and computer-assisted diagnostic system development.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 74 of its 148 references that are in the map. First posted on arXiv 14 Jul 2023.

Builds on

74 of 148 references.

References · 74

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  8. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  11. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  12. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
38 more
  1. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  2. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  3. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  4. 2019A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAIcited by 403
  5. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  6. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  7. 2018Bilinear Attention Networkscited by 277
  8. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  9. 2020COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Imagescited by 264
  10. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183
  11. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  12. 2021Explainable artificial intelligence (XAI) in deep learning-based medical image analysiscited by 147
  13. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  14. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 139
  15. 2017Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answeringcited by 117
  16. 2022LViT: Language meets Vision Transformer in Medical Image Segmentationcited by 110
  17. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  18. 2020COVID-19 Image Data Collection: Prospective Predictions Are the Futurecited by 100
  19. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Trainingcited by 96
  20. 2022RoentGen: Vision-Language Foundation Model for Chest X-ray Generationcited by 89
  21. 2022Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domainscited by 85
  22. 2022Improving the Factual Correctness of Radiology Report Generation with Semantic Rewardscited by 82
  23. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  24. 2021Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reportscited by 80
  25. 2017Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answeringcited by 79
  26. 2020Interpretability of machine learning based prediction models in healthcarecited by 75
  27. 2022Competence-based Multimodal Curriculum Learning for Medical Report Generationcited by 74
  28. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  29. 2021MMBERT: Multimodal BERT Pretraining for Improved Medical VQAcited by 68
  30. 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Studycited by 66
  31. 2022AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generationcited by 65
  32. 2019Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichmentcited by 60
  33. 2021Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articlescited by 58
  34. 2021Multiple Meta-model Quantifying for Medical Visual Question Answeringcited by 54
  35. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  36. 2020PixelHop++: A Small Successive-Subspace-Learning-Based (SSL-based) Model for Image Classificationcited by 37
  37. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  38. 2022Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Reviewcited by 37

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 3; 2026: 3.

13 papers.

Sort:

By area:

  1. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  2. 2023Medical Vision Language Pretraining: A surveycited by 24
  3. 2023A Survey on Image-text Multimodal Modelscited by 4
  4. 2024Enhancing medical vision-language contrastive learning via inter-matching relation modellingcited by 4
  5. 2025CLIMD: A Curriculum Learning Framework for Imbalanced Multimodal Diagnosiscited by 3
  6. 2026CT-Bench: A Benchmark for Multimodal Lesion Understanding in Computed Tomographycited by 2
  7. 2024Enhancing Scientific Figure Captioning Through Cross-modal Learningcited by 1
  8. 2024Advancing Histopathology-Based Breast Cancer Diagnosis: Insights into Multi-Modality and Explainabilitycited by 0
  9. 2024HyperMM : Robust Multimodal Learning with Varying-sized Inputscited by 0
  10. 2025Evaluating Strategies for Synthesizing Clinical Notes for Medical Multimodal AIcited by 0
  11. 2025Cross-modal ultra-scale learning with tri-modalities of renal biopsy images for glomerular multi-disease auxiliary diagnosiscited by 0
  12. 2026Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modelingcited by 0
1 more
  1. 2026PatTree: a novel approach for automated creation of multimodal, graph-based patient representations for medical classification taskscited by 0