# Vision Foundation Models in Medical Image Analysis: Advances and Challenges arXiv: https://arxiv.org/abs/2502.14584 ยท HTML: https://mlsanity.com/paper/2502.14584/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 52 of its 90 references that are in the map. - First posted on arXiv 20 Feb 2025. ## Builds on (most cited first, 30 of 52) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021) - [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021) - [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023) - [Domain Adaptation for Medical Image Analysis: A Survey](https://mlsanity.com/paper/2102.09508/) (2021) - [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023) - [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023) - [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024) - [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022) - [PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Model](https://mlsanity.com/paper/2208.11625/) (2022) - [CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No](https://mlsanity.com/paper/2308.12213/) (2023) - [Mixed Transformer U-Net For Medical Image Segmentation](https://mlsanity.com/paper/2111.04734/) (2021) - [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021) - [When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions](https://mlsanity.com/paper/2306.15546/) (2023) - [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024) - [Dual-Personalizing Adapter for Federated Foundation Models](https://mlsanity.com/paper/2403.19211/) (2024) - [A scoping review of transfer learning research on medical image analysis using ImageNet](https://mlsanity.com/paper/2004.13175/) (2020) - [Parameter-Efficient and Student-Friendly Knowledge Distillation](https://mlsanity.com/paper/2205.15308/) (2022) - [SwinMM: Masked Multi-view with Swin Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2307.12591/) (2023) - [TinySAM: Pushing the Envelope for Efficient Segment Anything Model](https://mlsanity.com/paper/2312.13789/) (2023) - [A Survey on Efficient Federated Learning Methods for Foundation Model Training](https://mlsanity.com/paper/2401.04472/) (2024) - [Foundation Models for Biomedical Image Segmentation: A Survey](https://mlsanity.com/paper/2401.07654/) (2024) - [FreeKD: Knowledge Distillation via Semantic Frequency Prompt](https://mlsanity.com/paper/2311.12079/) (2023) - [DeSAM: Decoupled Segment Anything Model for Generalizable Medical Image Segmentation](https://mlsanity.com/paper/2306.00499/) (2023) - [Segment Anything Model (SAM) for Radiation Oncology](https://mlsanity.com/paper/2306.11730/) (2023) - [Federated Generative Learning with Foundation Models](https://mlsanity.com/paper/2306.16064/) (2023) - [Learnable Prompting SAM-induced Knowledge Distillation for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2412.13742/) (2024) - [Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2304.06461/) (2023) - [Generic-to-Specific Distillation of Masked Autoencoders](https://mlsanity.com/paper/2302.14771/) (2023) - [UNetFormer: A Unified Vision Transformer Model and Pre-Training Framework for 3D Medical Image Segmentation](https://mlsanity.com/paper/2204.00631/) (2022) ## Built on by (most cited first, 9 of 9) - [Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directions](https://mlsanity.com/paper/2511.01284/) (2025) - [BleedOrigin: Dynamic Bleeding Source Localization in Endoscopic Submucosal Dissection via Dual-Stage Detection and Tracking](https://mlsanity.com/paper/2507.15094/) (2025) - [Advancements in Artificial Intelligence Applications for Cardiovascular Disease Research](https://mlsanity.com/paper/2506.03698/) (2025) - [Shaken or Stirred? An Analysis of MetaFormer's Token Mixing for Medical Imaging](https://mlsanity.com/paper/2510.05971/) (2025) - [VFM-ISRefiner: Towards Better Adapting Vision Foundation Models for Interactive Segmentation of Remote Sensing Images](https://mlsanity.com/paper/2512.00718/) (2025) - [Meta-Learning and Meta-Reinforcement Learning -- Tracing the Path towards DeepMind's Adaptive Agent](https://mlsanity.com/paper/2602.19837/) (2026) - [Prompting with the human-touch: evaluating model-sensitivity of foundation models for musculoskeletal CT segmentation](https://mlsanity.com/paper/2603.10541/) (2026) - [Toward Clinically Ready Foundation Models in Medical Image Analysis: Adaptation Mechanisms and Deployment Trade-offs](https://mlsanity.com/paper/2603.14271/) (2026) - [Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space](https://mlsanity.com/paper/2607.01689/) (2026) ## Abstract (arXiv) The rapid development of Vision Foundation Models (VFMs), particularly Vision Transformers (ViT) and Segment Anything Model (SAM), has sparked significant advances in the field of medical image analysis. These models have demonstrated exceptional capabilities in capturing long-range dependencies and achieving high generalization in segmentation tasks. However, adapting these large models to medical image analysis presents several challenges, including domain differences between medical and natural images, the need for efficient model adaptation strategies, and the limitations of small-scale medical datasets. This paper reviews the state-of-the-art research on the adaptation of VFMs to medical image segmentation, focusing on the challenges of domain adaptation, model compression, and federated learning. We discuss the latest developments in adapter-based improvements, knowledge distillation techniques, and multi-scale contextual feature modeling, and propose future directions to overcome these bottlenecks. Our analysis highlights the potential of VFMs, along with emerging methodologies such as federated learning and model compression, to revolutionize medical image analysis and enhance clinical applications. The goal of this work is to provide a comprehensive overview of current approaches and suggest key areas for future research that can drive the next wave of innovation in medical image segmentation.