# Self-Prompting Polyp Segmentation in Colonoscopy using Hybrid Yolo-SAM 2 Model

arXiv: https://arxiv.org/abs/2409.09484 · HTML: https://mlsanity.com/paper/2409.09484/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 14 Sep 2024.

## Builds on (most cited first, 27 of 27)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images](https://mlsanity.com/paper/1612.00799/) (2016)
- [ResUNet++: An Advanced Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1911.07067/) (2019)
- [UNeXt: MLP-based Rapid Medical Image Segmentation Network](https://mlsanity.com/paper/2203.04967/) (2022)
- [Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers](https://mlsanity.com/paper/2108.06932/) (2021)
- [Shallow Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2108.00882/) (2021)
- [SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation](https://mlsanity.com/paper/2308.08746/) (2023)
- [Video Polyp Segmentation: A Deep Learning Perspective](https://mlsanity.com/paper/2203.14291/) (2022)
- [Automatic Polyp Segmentation via Multi-scale Subtraction Network](https://mlsanity.com/paper/2108.05082/) (2021)
- [Progressively Normalized Self-Attention Network for Video Polyp Segmentation](https://mlsanity.com/paper/2105.08468/) (2021)
- [HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPS](https://mlsanity.com/paper/2101.07172/) (2021)
- [Polyp-SAM: Transfer SAM for Polyp Segmentation](https://mlsanity.com/paper/2305.00293/) (2023)
- [A multi-centre polyp detection and segmentation dataset for generalisability assessment](https://mlsanity.com/paper/2106.04463/) (2021)
- [UACANet: Uncertainty Augmented Context Attention for Polyp Segmentation](https://mlsanity.com/paper/2107.02368/) (2021)
- [Implicit Motion Handling for Video Camouflaged Object Detection](https://mlsanity.com/paper/2203.07363/) (2022)
- [Stepwise Feature Fusion: Local Guides Global](https://mlsanity.com/paper/2203.03635/) (2022)
- [TransNetR: Transformer-based Residual Network for Polyp Segmentation with Multi-Center Out-of-Distribution Testing](https://mlsanity.com/paper/2303.07428/) (2023)
- [SAM-Path: A Segment Anything Model for Semantic Segmentation in Digital Pathology](https://mlsanity.com/paper/2307.09570/) (2023)
- [Shifting More Attention to Breast Lesion Segmentation in Ultrasound Videos](https://mlsanity.com/paper/2310.01861/) (2023)
- [Unleashing the Potential of SAM2 for Biomedical Images and Videos: A Survey](https://mlsanity.com/paper/2408.12889/) (2024)
- [Polyp SAM 2: Advancing Zero shot Polyp Segmentation in Colorectal Cancer Detection](https://mlsanity.com/paper/2408.05892/) (2024)
- [Efficient Polyp Segmentation Via Integrity Learning](https://mlsanity.com/paper/2309.08234/) (2023)

## Built on by (most cited first, 6 of 6)

- [SAM2 for Image and Video Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2503.12781/) (2025)
- [Prompt Engineering in Segment Anything Model: Methodologies, Applications, and Emerging Challenges](https://mlsanity.com/paper/2507.09562/) (2025)
- [CrackESS: A Self-Prompting Crack Segmentation System for Edge Devices](https://mlsanity.com/paper/2412.07205/) (2024)
- [Advancements in Medical Image Classification through Fine-Tuning Natural Domain Foundation Models](https://mlsanity.com/paper/2505.19779/) (2025)
- [Towards Better Ultrasound Video Segmentation Foundation Model: An Empirical study on SAM2 Finetuning from Data Perspective](https://mlsanity.com/paper/2511.05731/) (2025)
- [Learning-based Hierarchical Tracheal Anatomy Understanding from Sparse Surgical Demonstration Annotations for Ultrasound Robots](https://mlsanity.com/paper/2607.22789/) (2026)

## Abstract (arXiv)

Early diagnosis and treatment of polyps during colonoscopy are essential for reducing the incidence and mortality of Colorectal Cancer (CRC). However, the variability in polyp characteristics and the presence of artifacts in colonoscopy images and videos pose significant challenges for accurate and efficient polyp detection and segmentation. This paper presents a novel approach to polyp segmentation by integrating the Segment Anything Model (SAM 2) with the YOLOv8 model. Our method leverages YOLOv8's bounding box predictions to autonomously generate input prompts for SAM 2, thereby reducing the need for manual annotations. We conducted exhaustive tests on five benchmark colonoscopy image datasets and two colonoscopy video datasets, demonstrating that our method exceeds state-of-the-art models in both image and video segmentation tasks. Notably, our approach achieves high segmentation accuracy using only bounding box annotations, significantly reducing annotation time and effort. This advancement holds promise for enhancing the efficiency and scalability of polyp detection in clinical settings https://github.com/sajjad-sh33/YOLO_SAM2.