mlsanityData to 7 Oct 2026

On the Challenges and Perspectives of Foundation Models for Medical Image Analysis

arXiv 2306.05705 ↗First posted 9 Jun 2023eess.IV cs.CVcited by 135

Not on the map. Why?

Abstract (arXiv)

This article discusses the opportunities, applications and future directions of large-scale pre-trained models, i.e., foundation models, for analyzing medical images. Medical foundation models have immense potential in solving a wide range of downstream tasks, as they can help to accelerate the development of accurate and robust models, reduce the large amounts of required labeled data, preserve the privacy and confidentiality of patient data. Specifically, we illustrate the "spectrum" of medical foundation models, ranging from general vision models, modality-specific models, to organ/task-specific models, highlighting their challenges, opportunities and applications. We also discuss how foundation models can be leveraged in downstream medical tasks to enhance the accuracy and efficiency of medical image analysis, leading to more precise diagnosis and treatment decisions.

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). Builds on 51 of its 88 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

51 of 88 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  6. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  7. 2023PaLM 2 Technical Reportcited by 1,154
  8. 2021Scaling Vision Transformerscited by 993
  9. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  10. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  11. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  12. 2021The Medical Segmentation Decathloncited by 620
  13. 2022TotalSegmentator: robust segmentation of 104 anatomical structures in CT imagescited by 597
  14. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  15. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  16. 2023Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentationcited by 485
  17. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  18. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 418
  19. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  20. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
  21. 2023Segment Anything Model for Medical Image Analysis: an Experimental Studycited by 291
  22. 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 282
  23. 2022Brain Imaging Generation with Latent Diffusion Modelscited by 267
  24. 2023Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imagingcited by 161
  25. 2020Multi-site fMRI Analysis Using Privacy-preserving Federated Learning and Domain Adaptation: ABIDE Resultscited by 122
  26. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  27. 2022ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation datasetcited by 104
  28. 2023STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-trainingcited by 96
  29. 2023UniverSeg: Universal Medical Image Segmentationcited by 90
  30. 2022RoentGen: Vision-Language Foundation Model for Chest X-ray Generationcited by 89
  31. 2022Video Polyp Segmentation: A Deep Learning Perspectivecited by 81
  32. 2023SAM on Medical Images: A Comprehensive Study on Three Prompt Modescited by 81
  33. 2021Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reportscited by 80
  34. 2023Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentationcited by 51
  35. 2020Self-supervised Feature Learning via Exploiting Multi-modal Data for Retinal Disease Diagnosiscited by 47
  36. 2023AIROGS: Artificial Intelligence for RObust Glaucoma Screening Challengecited by 35
  37. 2022Self-supervised Learning from 100 Million Medical Imagescited by 31
  38. 2023SAMM (Segment Any Medical Model): A 3D Slicer Integration to SAMcited by 31

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 40; 2025: 57; 2026: 25.

135 papers.

Sort:

By area:

  1. 2024Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directionscited by 97
  2. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  3. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
  4. 20233DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentationcited by 70
  5. 2024AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarkingcited by 64
  6. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  7. 2024Segment Anything in Medical Images and Videos: Benchmark and Deploymentcited by 42
  8. 2023CLIP in Medical Imaging: A Surveycited by 40
  9. 2024BrainSegFounder: Towards 3D Foundation Models for Neuroimage Segmentationcited by 36
  10. 2024Interactive Medical Image Segmentation: A Benchmark Dataset and Baselinecited by 30
  11. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  12. 2023MIS-FM: 3D Medical Image Segmentation using Foundation Models Pretrained on a Large-Scale Unannotated Datasetcited by 25
38 more
  1. 2023Dual-Reference Source-Free Active Domain Adaptation for Nasopharyngeal Carcinoma Tumor Segmentation across Multiple Hospitalscited by 25
  2. 20253D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Taskscited by 24
  3. 2024MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Modelscited by 21
  4. 2025Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imagingcited by 20
  5. 2025How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?cited by 19
  6. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 17
  7. 2024Benchmarking Pathology Foundation Models: Adaptation Strategies and Scenarioscited by 16
  8. 2023ZePT: Zero-Shot Pan-Tumor Segmentation via Query-Disentangling and Self-Promptingcited by 15
  9. 2025Towards a general-purpose foundation model for fMRI analysiscited by 15
  10. 2024CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentationcited by 14
  11. 2025BRISC: Annotated Dataset for Brain Tumor Segmentation and Classificationcited by 14
  12. 2024A Foundation Model for Brain Lesion Segmentation with Mixture of Modality Expertscited by 13
  13. 2024Towards a vision foundation model for comprehensive assessment of Cardiac MRIcited by 13
  14. 2025SegAnyPET: Universal Promptable Segmentation from Positron Emission Tomography Imagescited by 11
  15. 2023DIAS: A Dataset and Benchmark for Intracranial Artery Segmentation in DSA sequencescited by 10
  16. 2023Enhancing the Reliability of Segment Anything Model for Auto-Prompting Medical Image Segmentation with Uncertainty Rectificationcited by 8
  17. 2025Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Reviewcited by 8
  18. 2025MAUP: Training-free Multi-center Adaptive Uncertainty-aware Prompting for Cross-domain Few-shot Medical Image Segmentationcited by 8
  19. 2023When is a Foundation Model a Foundation Modelcited by 7
  20. 2023MMGPL: Multimodal Medical Data Analysis with Graph Prompt Learningcited by 7
  21. 2024Neural Radiance Fields in Medical Imaging: A Surveycited by 7
  22. 2024Cost-effective Instruction Learning for Pathology Vision and Language Analysiscited by 7
  23. 2024Tissue Concepts: supervised foundation models in computational pathologycited by 7
  24. 2025MIRAGE: Multimodal foundation model and benchmark for comprehensive retinal OCT image analysiscited by 7
  25. 2025Few-Shot, Now for Real: Medical VLMs Adaptation without Balanced Sets or Validationcited by 7
  26. 2025Improving Generalization of Medical Image Registration Foundation Modelcited by 6
  27. 2024Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinementcited by 5
  28. 2024OpenMEDLab: An Open-source Platform for Multi-modality Foundation Models in Medicinecited by 5
  29. 2024A Federated Learning-Friendly Approach for Parameter-Efficient Fine-Tuning of SAM in 3D Segmentationcited by 5
  30. 2024How Effective is Pre-training of Large Masked Autoencoders for Downstream Earth Observation Tasks?cited by 5
  31. 2025Prompt to Polyp: Medical Text-Conditioned Image Synthesis with Diffusion Modelscited by 5
  32. 2025Generalist versus Specialist Vision Foundation Models for Ocular Disease and Oculomicscited by 5
  33. 2025TeamPath: Building MultiModal Pathology Experts with Reasoning AI Copilotscited by 5
  34. 2024Probing the Efficacy of Federated Parameter-Efficient Fine-Tuning of Vision Transformers for Medical Image Classificationcited by 4
  35. 2024Semi-Supervised Fine-Tuning of Vision Foundation Models with Content-Style Decompositioncited by 4
  36. 2024Fusing Echocardiography Images and Medical Records for Continuous Patient Stratificationcited by 3
  37. 2025Fair Foundation Models for Medical Image Analysis: Challenges and Perspectivescited by 3
  38. 2025Can We Simplify Slide-level Fine-tuning of Pathology Foundation Models?cited by 3

50 most cited of 135 All 135 (JSON)