# Polyp-SAM: Transfer SAM for Polyp Segmentation

arXiv: https://arxiv.org/abs/2305.00293 · HTML: https://mlsanity.com/paper/2305.00293/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 29 Apr 2023.

## Builds on (most cited first, 13 of 13)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers](https://mlsanity.com/paper/2108.06932/) (2021)
- [Context-aware Cross-level Fusion Network for Camouflaged Object Detection](https://mlsanity.com/paper/2105.12555/) (2021)
- [SAM.MD: Zero-shot medical image segmentation capabilities of the Segment Anything Model](https://mlsanity.com/paper/2304.05396/) (2023)
- [Lesion-aware Dynamic Kernel for Polyp Segmentation](https://mlsanity.com/paper/2301.04904/) (2023)
- [SkinSAM: Empowering Skin Cancer Segmentation with Segment Anything Model](https://mlsanity.com/paper/2304.13973/) (2023)
- [Advancing Medical Imaging with Language Models: A Journey from N-grams to ChatGPT](https://mlsanity.com/paper/2304.04920/) (2023)
- [Duplex Contextual Relation Network for Polyp Segmentation](https://mlsanity.com/paper/2103.06725/) (2021)

## Built on by (most cited first, 30 of 64)

- [Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions](https://mlsanity.com/paper/2401.03495/) (2024)
- [A Comprehensive Survey on Segment Anything Model for Vision and Beyond](https://mlsanity.com/paper/2305.08196/) (2023)
- [Unleashing the Potential of SAM for Medical Adaptation via Hierarchical Decoding](https://mlsanity.com/paper/2403.18271/) (2024)
- [A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering](https://mlsanity.com/paper/2306.06211/) (2023)
- [How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model](https://mlsanity.com/paper/2404.09957/) (2024)
- [DarkSAM: Fooling Segment Anything Model to Segment Nothing](https://mlsanity.com/paper/2409.17874/) (2024)
- [Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2411.12814/) (2024)
- [Foundation Models for Biomedical Image Segmentation: A Survey](https://mlsanity.com/paper/2401.07654/) (2024)
- [Segment anything model 2: an application to 2D and 3D medical images](https://mlsanity.com/paper/2408.00756/) (2024)
- [Deep Interactive Segmentation of Medical Images: A Systematic Review and Taxonomy](https://mlsanity.com/paper/2311.13964/) (2023)
- [Data-Centric Foundation Models in Computational Healthcare: A Survey](https://mlsanity.com/paper/2401.02458/) (2024)
- [DeSAM: Decoupled Segment Anything Model for Generalizable Medical Image Segmentation](https://mlsanity.com/paper/2306.00499/) (2023)
- [On the Robustness of Segment Anything](https://mlsanity.com/paper/2305.16220/) (2023)
- [MedDINOv3: How to adapt vision foundation models for medical image segmentation?](https://mlsanity.com/paper/2509.02379/) (2025)
- [Segment Anything Model (SAM) for Radiation Oncology](https://mlsanity.com/paper/2306.11730/) (2023)
- [An Empirical Study on the Robustness of the Segment Anything Model (SAM)](https://mlsanity.com/paper/2305.06422/) (2023)
- [SAM-PARSER: Fine-tuning SAM Efficiently by Parameter Space Reconstruction](https://mlsanity.com/paper/2308.14604/) (2023)
- [Polyp-SAM++: Can A Text Guided SAM Perform Better for Polyp Segmentation?](https://mlsanity.com/paper/2308.06623/) (2023)
- [Towards Segment Anything Model (SAM) for Medical Image Segmentation: A Survey](https://mlsanity.com/paper/2305.03678/) (2023)
- [Text Augmented Spatial-aware Zero-shot Referring Image Segmentation](https://mlsanity.com/paper/2310.18049/) (2023)
- [Learning the What and How of Annotation in Video Object Segmentation](https://mlsanity.com/paper/2311.04414/) (2023)
- [Self-Sampling Meta SAM: Enhancing Few-shot Medical Image Segmentation with Meta-Learning](https://mlsanity.com/paper/2308.16466/) (2023)
- [SAFIRE: Segment Any Forged Image Region](https://mlsanity.com/paper/2412.08197/) (2024)
- [MedSAM-U: Uncertainty-Guided Auto Multi-Prompt Adaptation for Reliable MedSAM](https://mlsanity.com/paper/2409.00924/) (2024)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [Full-dose Whole-body PET Synthesis from Low-dose PET Using High-efficiency Denoising Diffusion Probabilistic Model: PET Consistency Model](https://mlsanity.com/paper/2308.13072/) (2023)
- [mathrmSAMMed: A medical image annotation framework based on large vision model](https://mlsanity.com/paper/2307.05617/) (2023)
- [A Survey on Deep Learning for Polyp Segmentation: Techniques, Challenges and Future Trends](https://mlsanity.com/paper/2311.18373/) (2023)
- [When SAM Meets Sonar Images](https://mlsanity.com/paper/2306.14109/) (2023)
- [EviPrompt: A Training-Free Evidential Prompt Generation Method for Segment Anything Model in Medical Images](https://mlsanity.com/paper/2311.06400/) (2023)

## Abstract (arXiv)

Colon polyps are considered important precursors for colorectal cancer. Automatic segmentation of colon polyps can significantly reduce the misdiagnosis of colon cancer and improve physician annotation efficiency. While many methods have been proposed for polyp segmentation, training large-scale segmentation networks with limited colonoscopy data remains a challenge. Recently, the Segment Anything Model (SAM) has recently gained much attention in both natural and medical image segmentation. SAM demonstrates superior performance in several image benchmarks and therefore shows great potential for medical image segmentation. In this study, we propose Poly-SAM, a finetuned SAM model for polyp segmentation, and compare its performance to several state-of-the-art polyp segmentation models. We also compare two transfer learning strategies of SAM with and without finetuning its encoders. Evaluated on five public datasets, our Polyp-SAM achieves state-of-the-art performance on two datasets and impressive performance on three datasets, with dice scores all above 88%. This study demonstrates the great potential of adapting SAM to medical image segmentation tasks. We plan to release the code and model weights for this paper at: https://github.com/ricklisz/Polyp-SAM.