# ASPS: Augmented Segment Anything Model for Polyp Segmentation

arXiv: https://arxiv.org/abs/2407.00718 · HTML: https://mlsanity.com/paper/2407.00718/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 30 Jun 2024.

## Builds on (most cited first, 29 of 29)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [Revisiting Batch Normalization For Practical Domain Adaptation](https://mlsanity.com/paper/1603.04779/) (2016)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation](https://mlsanity.com/paper/2209.08575/) (2022)
- [Learning Confidence for Out-of-Distribution Detection in Neural Networks](https://mlsanity.com/paper/1802.04865/) (2018)
- [A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images](https://mlsanity.com/paper/1612.00799/) (2016)
- [EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything](https://mlsanity.com/paper/2312.00863/) (2023)
- [MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation](https://mlsanity.com/paper/2212.01322/) (2022)
- [Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers](https://mlsanity.com/paper/2108.06932/) (2021)
- [Shallow Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2108.00882/) (2021)
- [SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation](https://mlsanity.com/paper/2308.08746/) (2023)
- [Fast Vision Transformers with HiLo Attention](https://mlsanity.com/paper/2205.13213/) (2022)
- [Automatic Polyp Segmentation via Multi-scale Subtraction Network](https://mlsanity.com/paper/2108.05082/) (2021)
- [Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train](https://mlsanity.com/paper/2306.16741/) (2023)
- [Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning](https://mlsanity.com/paper/2312.11420/) (2023)
- [UACANet: Uncertainty Augmented Context Attention for Polyp Segmentation](https://mlsanity.com/paper/2107.02368/) (2021)
- [Can SAM Segment Polyps?](https://mlsanity.com/paper/2304.07583/) (2023)
- [Stepwise Feature Fusion: Local Guides Global](https://mlsanity.com/paper/2203.03635/) (2022)
- [How to Efficiently Adapt Large Segmentation Model(SAM) to Medical Images](https://mlsanity.com/paper/2306.13731/) (2023)
- [Beyond Adapting SAM: Towards End-to-End Ultrasound Image Segmentation via Auto Prompting](https://mlsanity.com/paper/2309.06824/) (2023)
- [Lesion-aware Dynamic Kernel for Polyp Segmentation](https://mlsanity.com/paper/2301.04904/) (2023)
- [Context-Aware Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/2003.04010/) (2020)
- [SAM-Path: A Segment Anything Model for Semantic Segmentation in Digital Pathology](https://mlsanity.com/paper/2307.09570/) (2023)
- [Duplex Contextual Relation Network for Polyp Segmentation](https://mlsanity.com/paper/2103.06725/) (2021)
- [On the Out of Distribution Robustness of Foundation Models in Medical Image Segmentation](https://mlsanity.com/paper/2311.11096/) (2023)

## Built on by (most cited first, 12 of 12)

- [Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2411.12814/) (2024)
- [Frontiers in Intelligent Colonoscopy](https://mlsanity.com/paper/2410.17241/) (2024)
- [Polyp SAM 2: Advancing Zero shot Polyp Segmentation in Colorectal Cancer Detection](https://mlsanity.com/paper/2408.05892/) (2024)
- [CoSAM: Self-Correcting SAM for Domain Generalization in 2D Medical Image Segmentation](https://mlsanity.com/paper/2411.10136/) (2024)
- [A Physics-Informed Deep Learning Model for MRI Brain Motion Correction](https://mlsanity.com/paper/2502.09296/) (2025)
- [PraNet-V2: Dual-Supervised Reverse Attention for Medical Image Segmentation](https://mlsanity.com/paper/2504.10986/) (2025)
- [Fully Automated SAM for Single-source Domain Generalization in Medical Image Segmentation](https://mlsanity.com/paper/2507.17281/) (2025)
- [Feature-EndoGaussian: Feature Distilled Gaussian Splatting in Surgical Deformable Scene Reconstruction](https://mlsanity.com/paper/2503.06161/) (2025)
- [APSeg: Auto-Prompt Model with Acquired and Injected Knowledge for Nuclear Instance Segmentation and Classification](https://mlsanity.com/paper/2504.02222/) (2025)
- [Improved Segmentation of Polyps and Visual Explainability Analysis](https://mlsanity.com/paper/2509.18159/) (2025)
- [IP-SAM: Rethinking Prompt-Conditioned Segmentation for Prompt-Absent Deployment](https://mlsanity.com/paper/2603.27250/) (2026)
- [ReG-SAM: Reference Graph-Driven SAM for 2D Foundational Vessel Segmentation](https://mlsanity.com/paper/2609.31160/) (2026)

## Abstract (arXiv)

Polyp segmentation plays a pivotal role in colorectal cancer diagnosis. Recently, the emergence of the Segment Anything Model (SAM) has introduced unprecedented potential for polyp segmentation, leveraging its powerful pre-training capability on large-scale datasets. However, due to the domain gap between natural and endoscopy images, SAM encounters two limitations in achieving effective performance in polyp segmentation. Firstly, its Transformer-based structure prioritizes global and low-frequency information, potentially overlooking local details, and introducing bias into the learned features. Secondly, when applied to endoscopy images, its poor out-of-distribution (OOD) performance results in substandard predictions and biased confidence output. To tackle these challenges, we introduce a novel approach named Augmented SAM for Polyp Segmentation (ASPS), equipped with two modules: Cross-branch Feature Augmentation (CFA) and Uncertainty-guided Prediction Regularization (UPR). CFA integrates a trainable CNN encoder branch with a frozen ViT encoder, enabling the integration of domain-specific knowledge while enhancing local features and high-frequency details. Moreover, UPR ingeniously leverages SAM's IoU score to mitigate uncertainty during the training procedure, thereby improving OOD performance and domain generalization. Extensive experimental results demonstrate the effectiveness and utility of the proposed method in improving SAM's performance in polyp segmentation. Our code is available at https://github.com/HuiqianLi/ASPS.