# Towards Segment Anything Model (SAM) for Medical Image Segmentation: A Survey

arXiv: https://arxiv.org/abs/2305.03678 · HTML: https://mlsanity.com/paper/2305.03678/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 70 references that are in the map.
- First posted on arXiv 5 May 2023.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation](https://mlsanity.com/paper/1908.10454/) (2019)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications](https://mlsanity.com/paper/2304.05750/) (2023)
- [SAM on Medical Images: A Comprehensive Study on Three Prompt Modes](https://mlsanity.com/paper/2305.00035/) (2023)
- [Learning with Limited Annotations: A Survey on Deep Semi-Supervised Learning for Medical Image Segmentation](https://mlsanity.com/paper/2207.14191/) (2022)
- [SAM Struggles in Concealed Scenes -- Empirical Study on Segment Anything](https://mlsanity.com/paper/2304.06022/) (2023)
- [CycleMix: A Holistic Strategy for Medical Image Segmentation from Scribble Supervision](https://mlsanity.com/paper/2203.01475/) (2022)
- [Scribble-Supervised Medical Image Segmentation via Dual-Branch Network and Dynamically Mixed Pseudo Labels Supervision](https://mlsanity.com/paper/2203.02106/) (2022)
- [Polyp-SAM: Transfer SAM for Polyp Segmentation](https://mlsanity.com/paper/2305.00293/) (2023)
- [SAM.MD: Zero-shot medical image segmentation capabilities of the Segment Anything Model](https://mlsanity.com/paper/2304.05396/) (2023)
- [Can SAM Segment Polyps?](https://mlsanity.com/paper/2304.07583/) (2023)
- [Input Augmentation with SAM: Boosting Medical Image Segmentation with Segmentation Foundation Model](https://mlsanity.com/paper/2304.11332/) (2023)

## Built on by (most cited first, 21 of 21)

- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering](https://mlsanity.com/paper/2306.06211/) (2023)
- [Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks](https://mlsanity.com/paper/2312.02366/) (2023)
- [How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model](https://mlsanity.com/paper/2404.09957/) (2024)
- [Foundation Models for Biomedical Image Segmentation: A Survey](https://mlsanity.com/paper/2401.07654/) (2024)
- [SAM2 for Image and Video Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2503.12781/) (2025)
- [No time to train! Training-Free Reference-Based Instance Segmentation](https://mlsanity.com/paper/2507.02798/) (2025)
- [SAMed-2: Selective Memory Enhanced Medical Segment Anything Model](https://mlsanity.com/paper/2507.03698/) (2025)
- [Assessing Test-time Variability for Interactive 3D Medical Image Segmentation with Diverse Point Prompts](https://mlsanity.com/paper/2311.07806/) (2023)
- [Segment Anything for Videos: A Systematic Survey](https://mlsanity.com/paper/2408.08315/) (2024)
- [Few-Shot Adaptation of Training-Free Foundation Model for 3D Medical Image Segmentation](https://mlsanity.com/paper/2501.09138/) (2025)
- [Prompt Engineering in Segment Anything Model: Methodologies, Applications, and Emerging Challenges](https://mlsanity.com/paper/2507.09562/) (2025)
- [MedLSAM: Localize and Segment Anything Model for 3D CT Images](https://mlsanity.com/paper/2306.14752/) (2023)
- [Segment Anything for comprehensive analysis of grapevine cluster architecture and berry properties](https://mlsanity.com/paper/2403.12935/) (2024)
- [SynthFM: Training Modality-agnostic Foundation Models for Medical Image Segmentation without Real Medical Data](https://mlsanity.com/paper/2504.08177/) (2025)
- [AdLER: Adversarial Training with Label Error Rectification for One-Shot Medical Image Segmentation](https://mlsanity.com/paper/2309.00971/) (2023)
- [SARTM: Segment Any RGB Thermal Model with Language aided Distillation](https://mlsanity.com/paper/2505.01950/) (2025)
- [AutoMiSeg: Automatic Medical Image Segmentation via Test-Time Adaptation of Foundation Models](https://mlsanity.com/paper/2505.17931/) (2025)
- [Semantic Segmentation of iPS Cells: Case Study on Model Complexity in Biomedical Imaging](https://mlsanity.com/paper/2507.21608/) (2025)
- [Enhancing MedSAM with a Lightweight Box Predictor for Medical Image Segmentation](https://mlsanity.com/paper/2606.04705/) (2026)
- [NeuroRefiner: Morphology-Aware Multi-Agent Refinement for 3D Fluorescence Microscopy Neuron Segmentation](https://mlsanity.com/paper/2608.09636/) (2026)

## Abstract (arXiv)

Due to the flexibility of prompting, foundation models have become the dominant force in the domains of natural language processing and image generation. With the recent introduction of the Segment Anything Model (SAM), the prompt-driven paradigm has entered the realm of image segmentation, bringing with a range of previously unexplored capabilities. However, it remains unclear whether it can be applicable to medical image segmentation due to the significant differences between natural images and medical images.In this work, we summarize recent efforts to extend the success of SAM to medical image segmentation tasks, including both empirical benchmarking and methodological adaptations, and discuss potential future directions for SAM in medical image segmentation. Although directly applying SAM to medical image segmentation cannot obtain satisfying performance on multi-modal and multi-target medical datasets, many insights are drawn to guide future research to develop foundation models for medical image analysis. To facilitate future research, we maintain an active repository that contains up-to-date paper list and open-source project summary at https://github.com/YichiZhang98/SAM4MIS.