mlsanityData to 7 Oct 2026

Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions

arXiv 2412.02621 ↗First posted 3 Dec 2024cs.AI cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Recent advancements in deep learning have significantly revolutionized the field of clinical diagnosis and treatment, offering novel approaches to improve diagnostic precision and treatment efficacy across diverse clinical domains, thus driving the pursuit of precision medicine. The growing availability of multi-organ and multimodal datasets has accelerated the development of large-scale Medical Multimodal Foundation Models (MMFMs). These models, known for their strong generalization capabilities and rich representational power, are increasingly being adapted to address a wide range of clinical tasks, from early diagnosis to personalized treatment strategies. This review offers a comprehensive analysis of recent developments in MMFMs, focusing on three key aspects: datasets, model architectures, and clinical applications. We also explore the challenges and opportunities in optimizing multimodal representations and discuss how these advancements are shaping the future of healthcare by enabling improved patient outcomes and more efficient clinical workflows.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 104 of its 182 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

104 of 182 references.

References · 104

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2023Segment Anythingcited by 8,158
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2017Mask R-CNNcited by 7,259
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2022Generative Adversarial Networkscited by 5,603
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  6. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  7. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  8. 2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 823
  9. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  11. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  12. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  13. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  14. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  15. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  16. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  17. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  18. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  19. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  20. 20183D MRI brain tumor segmentation using autoencoder regularizationcited by 324
  21. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  22. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 308
  23. 2023Towards Generalist Biomedical AIcited by 286
  24. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  25. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  26. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  27. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  28. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  29. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  30. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 160
  31. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 157
  32. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  33. 2024Medical SAM 2: Segment medical images as video via Segment Anything Model 2cited by 151
  34. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 149
  35. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  36. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  37. 2023On the Challenges and Perspectives of Foundation Models for Medical Image Analysiscited by 135
  38. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128

50 most cited of 104 All 104 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 7.

17 papers.

Sort:

By area:

  1. 2025Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Reviewcited by 8
  2. 2025Med-GLIP: Advancing Medical Language-Image Pre-training with Large-scale Grounded Datasetcited by 7
  3. 2025Applications of Large Models in Medicinecited by 6
  4. 2025HeartcareGPT: A Unified Multimodal ECG Suite for Dual Signal-Image Modeling and Understandingcited by 3
  5. 2025MedIQA: A Scalable Foundation Model for Prompt-Driven Medical Image Quality Assessmentcited by 2
  6. 2025Multimodal Foundation Models for Early Disease Detectioncited by 1
  7. 2025MedInsightBench: Evaluating Medical Analytics Agents Through Multi-Step Insight Discovery in Multimodal Medical Datacited by 1
  8. 2026Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoningcited by 1
  9. 2025CLAPS: A CLIP-Unified Auto-Prompt Segmentation for Multi-Modal Retinal Imagingcited by 0
  10. 2025Evaluating Fundus-Specific Foundation Models for Diabetic Macular Edema Detectioncited by 0
  11. 2025Can Large Language Models Function as Qualified Pediatricians? A Systematic Evaluation in Real-World Clinical Contextscited by 0
  12. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
5 more
  1. 2026Imaging-anchored Multiomics in Cardiovascular Disease: Integrating Cardiac Imaging, Bulk, Single-cell, and Spatial Transcriptomicscited by 0
  2. 2026Foundation Models for Medical Imaging: Status, Challenges, and Directionscited by 0
  3. 2026Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRIcited by 0
  4. 2026In-Context Learning for Wound Classification with Small Multimodal Language Modelscited by 0
  5. 2026Protecting patient privacy in clinical foundation models: Technical and legal perspectivescited by 0