mlsanityData to 14 Oct 2026

Are Natural Domain Foundation Models Useful for Medical Image Classification?

arXiv 2310.19522 ↗First posted 30 Oct 2023cs.CVcited by 46

Not on the map. Why?

Abstract (arXiv)

The deep learning field is converging towards the use of general foundation models that can be easily adapted for diverse tasks. While this paradigm shift has become common practice within the field of natural language processing, progress has been slower in computer vision. In this paper we attempt to address this issue by investigating the transferability of various state-of-the-art foundation models to medical image classification tasks. Specifically, we evaluate the performance of five foundation models, namely SAM, SEEM, DINOv2, BLIP, and OpenCLIP across four well-established medical imaging datasets. We explore different training settings to fully harness the potential of these models. Our study shows mixed results. DINOv2 consistently outperforms the standard practice of ImageNet pretraining. However, other foundation models failed to consistently beat this established baseline indicating limitations in their transferability to medical image classification tasks.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 34 of its 52 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

34 of 52 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2023Segment Anythingcited by 8,234
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
22 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  5. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  6. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  7. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  8. 2023Segment Everything Everywhere All at Oncecited by 548
  9. 2023Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentationcited by 490
  10. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  11. 2023Segment Anything in Medical Imagescited by 227
  12. 2022Focal Modulation Networkscited by 186
  13. 2023Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imagingcited by 161
  14. 2023SAM on Medical Images: A Comprehensive Study on Three Prompt Modescited by 82
  15. 2022What Makes Transfer Learning Work For Medical Images: Feature Reuse & Other Factorscited by 61
  16. 2021Is it Time to Replace CNNs with Transformers for Medical Images?cited by 60
  17. 2023Can SAM Segment Polyps?cited by 59
  18. 2023Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentationcited by 51
  19. 2023Computer-Vision Benchmark Segment-Anything Model (SAM) in Medical Images: Accuracy in 12 Datasetscited by 45
  20. 2023Learnable Ophthalmology SAMcited by 21
  21. 2023Towards General Purpose Medical AI: Continual Learning Medical Foundation Modelcited by 14
  22. 2023Pretrained ViTs Yield Versatile Representations For Medical Imagescited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 18; 2026: 10.

46 papers.

Sort:

By area:

  1. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  2. 2024Comparative Analysis of ImageNet Pre-Trained Deep Learning Models and DINOv2 in Medical Imaging Classificationcited by 14
  3. 2024Towards a vision foundation model for comprehensive assessment of Cardiac MRIcited by 14
  4. 2024Medical Slice Transformer: Improved Diagnosis and Explainability on 3D Medical Images with DINOv2cited by 13
  5. 2024FACMIC: Federated Adaptative CLIP Model for Medical Image Classificationcited by 7
  6. 2025General Methods Make Great Domain-specific Foundation Models: A Case-study on Fetal Ultrasoundcited by 7
  7. 2025Boosting multi-demographic federated learning for chest radiograph analysis using general-purpose self-supervised representationscited by 6
  8. 2025Adapting Vision-Language Foundation Model for Next Generation Medical Ultrasound Image Analysiscited by 6
  9. 2024A Federated Learning-Friendly Approach for Parameter-Efficient Fine-Tuning of SAM in 3D Segmentationcited by 5
  10. 2024How Effective is Pre-training of Large Masked Autoencoders for Downstream Earth Observation Tasks?cited by 5
  11. 2025FAA-CLIP: Federated Adversarial Adaptation of CLIPcited by 5
  12. 2024Swin transformers are robust to distribution and concept drift in endoscopy-based longitudinal rectal cancer assessmentcited by 4
34 more
  1. 2024Graph Neural Networks: A suitable Alternative to MLPs in Latent 3D Medical Image Classification?cited by 4
  2. 2025Order Matters: On Parameter-Efficient Image-to-Video Probing for Recognizing Nearly Symmetric Actionscited by 3
  3. 2025DinoAtten3D: Slice-Level Attention Aggregation of DinoV2 for 3D Brain MRI Anomaly Classificationcited by 3
  4. 2024Masked Image Modelling for retinal OCT understandingcited by 2
  5. 2024Can language-guided unsupervised adaptation improve medical image classification using unpaired images and texts?cited by 2
  6. 2025Pre-trained Models Succeed in Medical Imaging with Representation Similarity Degradationcited by 2
  7. 2025Conformal uncertainty quantification to evaluate predictive fairness of foundation AI model for skin lesion classes across patient demographicscited by 2
  8. 2025DINO-MX: A Modular & Flexible Framework for Self-Supervised Learningcited by 2
  9. 2025Interpretable Similarity of Synthetic Image Utilitycited by 2
  10. 2024General Vision Encoder Features as Guidance in Medical Image Registrationcited by 1
  11. 2024The Inter-Intra Modal Measure: A Predictive Lens on Fine-Tuning Outcomes in Vision-Language Modelscited by 1
  12. 2024DINOv2 Rocks Geological Image Analysis: Classification, Segmentation, and Interpretabilitycited by 1
  13. 2024Medical Image Segmentation with SAM-generated Annotationscited by 1
  14. 2025Advancements in Medical Image Classification through Fine-Tuning Natural Domain Foundation Modelscited by 1
  15. 2025Deep Modeling and Optimization of Medical Image Classificationcited by 1
  16. 2025CardioBench: Do Echocardiography Foundation Models Generalize Beyond the Lab?cited by 1
  17. 2025TomoGraphView: 3D Medical Image Classification with Omnidirectional Slice Representations and Graph Neural Networkscited by 1
  18. 2026A Hierarchical Benchmark of Foundation Models for Dermatologycited by 1
  19. 2026Attend what matters: Leveraging vision foundational models for breast cancer classification using mammogramscited by 1
  20. 2026Foundation VAEs for 3D CT Reconstruction, Augmentation, and Generationcited by 1
  21. 2024Using Few-Shot Learning to Classify Primary Lung Cancer and Other Malignancy with Lung Metastasis in Cytological Imaging via Endobronchial Ultrasound Procedurescited by 0
  22. 2024Human-in-the-Loop Segmentation of Multi-species Coral Imagerycited by 0
  23. 2024Panoptic Segmentation of Mammograms with Text-To-Image Diffusion Modelcited by 0
  24. 2025Comparative Analysis of Pre-trained Deep Learning Models and DINOv2 for Cushing's Syndrome Diagnosis in Facial Analysiscited by 0
  25. 2025Efficient Self-Supervised Adaptation for Medical Image Analysiscited by 0
  26. 2025AI-Assisted Colonoscopy: Polyp Detection and Segmentation using Foundation Modelscited by 0
  27. 2025Learning What Helps: Task-Aligned Context Selection for Vision Taskscited by 0
  28. 2026Foundation Models for Medical Imaging: Status, Challenges, and Directionscited by 0
  29. 2026FederatedFactory: Generative One-Shot Learning for Extremely Non-IID Distributed Scenarioscited by 0
  30. 2026Are Natural-Domain Foundation Models Effective for Accelerated Cardiac MRI Reconstruction?cited by 0
  31. 2026Benchmarking transferability of SSL pretraining to same and different modality segmentation taskscited by 0
  32. 2026PEFT-MedSAM: Efficient Fine-Tuning of Medical Foundation Models for Explainable Skin Lesion Segmentationcited by 0
  33. 2026Localization Lens for Improving Medical Vision-Language Modelscited by 0
  34. 2026Beyond the Encoder: Object-Centric Representations Emerging from Segmentation Foundation Modelscited by 0