# How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model

arXiv: https://arxiv.org/abs/2404.09957 · HTML: https://mlsanity.com/paper/2404.09957/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 66 of its 114 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Builds on (most cited first, 30 of 66)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentation](https://mlsanity.com/paper/2001.06535/) (2020)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment](https://mlsanity.com/paper/2312.12148/) (2023)
- [U-Net and its variants for medical image segmentation: theory and applications](https://mlsanity.com/paper/2011.01118/) (2020)
- [Efficient parametrization of multi-domain deep neural networks](https://mlsanity.com/paper/1803.10082/) (2018)
- [Medical Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2009.13120/) (2020)
- [Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging](https://mlsanity.com/paper/2304.04155/) (2023)
- [On the Effectiveness of Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2211.15583/) (2022)

## Built on by (most cited first, 30 of 32)

- [Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?](https://mlsanity.com/paper/2411.03670/) (2024)
- [Segment anything model 2: an application to 2D and 3D medical images](https://mlsanity.com/paper/2408.00756/) (2024)
- [Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts](https://mlsanity.com/paper/2412.04220/) (2024)
- [MRI-CORE: A Foundation Model for Magnetic Resonance Imaging](https://mlsanity.com/paper/2506.12186/) (2025)
- [Sam2Rad: A Segmentation Model for Medical Images with Learnable Prompts](https://mlsanity.com/paper/2409.06821/) (2024)
- [MedicoSAM: Robust Improvement of SAM for Medical Imaging](https://mlsanity.com/paper/2501.11734/) (2025)
- [Are Vision Foundation Models Ready for Out-of-the-Box Medical Image Registration?](https://mlsanity.com/paper/2507.11569/) (2025)
- [Decipher-MR: A Vision-Language Foundation Model for 3D MRI Representations](https://mlsanity.com/paper/2509.21249/) (2025)
- [Quantifying the Limits of Segmentation Foundation Models: Modeling Challenges in Segmenting Tree-Like and Low-Contrast Objects](https://mlsanity.com/paper/2412.04243/) (2024)
- [Automated Muscle and Fat Segmentation in Computed Tomography for Comprehensive Body Composition Analysis](https://mlsanity.com/paper/2502.09779/) (2025)
- [SurgiSAM2: Fine-tuning a foundational model for surgical video anatomy segmentation and detection](https://mlsanity.com/paper/2503.03942/) (2025)
- [Universal and Transferable Attacks on Pathology Foundation Models](https://mlsanity.com/paper/2510.16660/) (2025)
- [Auditing Significance, Metric Choice, and Demographic Fairness in Medical AI Challenges](https://mlsanity.com/paper/2512.19091/) (2025)
- [EP-SAM: Weakly Supervised Histopathology Segmentation via Enhanced Prompt with Segment Anything](https://mlsanity.com/paper/2410.13621/) (2024)
- [Parameter Efficient Fine-Tuning of Segment Anything Model for Biomedical Imaging](https://mlsanity.com/paper/2502.00418/) (2025)
- [SegmentAnyMuscle: A universal muscle segmentation model across different locations in MRI](https://mlsanity.com/paper/2506.22467/) (2025)
- [Landmark Detection for Medical Images using a General-purpose Segmentation Model](https://mlsanity.com/paper/2507.11551/) (2025)
- [Tumor-anchored deep feature random forests for out-of-distribution detection in lung cancer segmentation](https://mlsanity.com/paper/2512.08216/) (2025)
- [From pre-training to downstream performance: Does domain-specific pre-training make sense?](https://mlsanity.com/paper/2605.08819/) (2026)
- [Optimized Vessel Segmentation: A Structure-Agnostic Approach with Small Vessel Enhancement and Morphological Correction](https://mlsanity.com/paper/2411.15251/) (2024)
- [Revisiting Learning Rate Control](https://mlsanity.com/paper/2507.01724/) (2025)
- [Assessment of Using Synthetic Data in Brain Tumor Segmentation](https://mlsanity.com/paper/2508.11922/) (2025)
- [Random forest-based out-of-distribution detection for robust lung cancer segmentation](https://mlsanity.com/paper/2508.19112/) (2025)
- [MAE-SAM2: Mask Autoencoder-Enhanced SAM2 for Clinical Retinal Vascular Leakage Segmentation](https://mlsanity.com/paper/2509.10554/) (2025)
- [Intelligent Communication Mixture-of-Experts Boosted-Medical Image Segmentation Foundation Model](https://mlsanity.com/paper/2510.17684/) (2025)
- [Towards Better Ultrasound Video Segmentation Foundation Model: An Empirical study on SAM2 Finetuning from Data Perspective](https://mlsanity.com/paper/2511.05731/) (2025)
- [SAMora: Enhancing SAM through Hierarchical Self-Supervised Pre-Training for Medical Images](https://mlsanity.com/paper/2511.08626/) (2025)
- [Harmonizing Generalization and Specialization: Uncertainty-Informed Collaborative Learning for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2512.13101/) (2025)
- [Spectral Rectification for Parameter-Efficient Adaptation of Foundation Models in Colonoscopy Depth Estimation](https://mlsanity.com/paper/2603.15374/) (2026)
- [Amped: Adaptive Multi-stage Non-edge Pruning for Edge Detection](https://mlsanity.com/paper/2603.27661/) (2026)

## Abstract (arXiv)

Automated segmentation is a fundamental medical image analysis task, which enjoys significant advances due to the advent of deep learning. While foundation models have been useful in natural language processing and some vision tasks for some time, the foundation model developed with image segmentation in mind - Segment Anything Model (SAM) - has been developed only recently and has shown similar promise. However, there are still no systematic analyses or "best-practice" guidelines for optimal fine-tuning of SAM for medical image segmentation. This work summarizes existing fine-tuning strategies with various backbone architectures, model components, and fine-tuning algorithms across 18 combinations, and evaluates them on 17 datasets covering all common radiology modalities. Our study reveals that (1) fine-tuning SAM leads to slightly better performance than previous segmentation methods, (2) fine-tuning strategies that use parameter-efficient learning in both the encoder and decoder are superior to other strategies, (3) network architecture has a small impact on final performance, (4) further training SAM with self-supervised learning can improve final model performance. We also demonstrate the ineffectiveness of some methods popular in the literature and further expand our experiments into few-shot and prompt-based settings. Lastly, we released our code and MRI-specific fine-tuned weights, which consistently obtained superior performance over the original SAM, at https://github.com/mazurowski-lab/finetune-SAM.