# RetSeg: Retention-based Colorectal Polyps Segmentation Network

arXiv: https://arxiv.org/abs/2310.05446 · HTML: https://mlsanity.com/paper/2310.05446/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 77 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images](https://mlsanity.com/paper/1612.00799/) (2016)
- [Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers](https://mlsanity.com/paper/2108.06932/) (2021)
- [Natural Language Processing: State of The Art, Current Trends and Challenges](https://mlsanity.com/paper/1708.05148/) (2017)
- [RMT: Retentive Networks Meet Vision Transformers](https://mlsanity.com/paper/2309.11523/) (2023)
- [Polyp-SAM: Transfer SAM for Polyp Segmentation](https://mlsanity.com/paper/2305.00293/) (2023)
- [TGANet: Text-guided attention for improved polyp segmentation](https://mlsanity.com/paper/2205.04280/) (2022)
- [Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions](https://mlsanity.com/paper/2305.10435/) (2023)
- [NeoUNet: Towards accurate colon polyp segmentation and neoplasm detection](https://mlsanity.com/paper/2107.05023/) (2021)
- [A Comprehensive Study on Colorectal Polyp Segmentation with ResUNet++, Conditional Random Field and Test-Time Augmentation](https://mlsanity.com/paper/2107.12435/) (2021)
- [DuAT: Dual-Aggregation Transformer Network for Medical Image Segmentation](https://mlsanity.com/paper/2212.11677/) (2022)
- [TransResU-Net: Transformer based ResU-Net for Real-Time Colonoscopy Polyp Segmentation](https://mlsanity.com/paper/2206.08985/) (2022)
- [Self-attention in Vision Transformers Performs Perceptual Grouping, Not Attention](https://mlsanity.com/paper/2303.01542/) (2023)
- [Transformers: "The End of History" for NLP?](https://mlsanity.com/paper/2105.00813/) (2021)
- [Optimizing Vision Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2210.08066/) (2022)
- [ESFPNet: efficient deep learning architecture for real-time lesion segmentation in autofluorescence bronchoscopic video](https://mlsanity.com/paper/2207.07759/) (2022)
- [Y-Net: A deep Convolutional Neural Network for Polyp Detection](https://mlsanity.com/paper/1806.01907/) (2018)
- [Convolutional Neural Network (CNN) vs Vision Transformer (ViT) for Digital Holography](https://mlsanity.com/paper/2108.09147/) (2021)
- [Vision Transformers in Medical Imaging: A Review](https://mlsanity.com/paper/2211.10043/) (2022)
- [DilatedSegNet: A Deep Dilated Segmentation Network for Polyp Segmentation](https://mlsanity.com/paper/2210.13595/) (2022)

## Built on by (most cited first, 1 of 1)

- [A Survey of Retentive Network](https://mlsanity.com/paper/2506.06708/) (2025)

## Abstract (arXiv)

Vision Transformers (ViTs) have revolutionized medical imaging analysis, showcasing superior efficacy compared to conventional Convolutional Neural Networks (CNNs) in vital tasks such as polyp classification, detection, and segmentation. Leveraging attention mechanisms to focus on specific image regions, ViTs exhibit contextual awareness in processing visual data, culminating in robust and precise predictions, even for intricate medical images. Moreover, the inherent self-attention mechanism in Transformers accommodates varying input sizes and resolutions, granting an unprecedented flexibility absent in traditional CNNs. However, Transformers grapple with challenges like excessive memory usage and limited training parallelism due to self-attention, rendering them impractical for real-time disease detection on resource-constrained devices. In this study, we address these hurdles by investigating the integration of the recently introduced retention mechanism into polyp segmentation, introducing RetSeg, an encoder-decoder network featuring multi-head retention blocks. Drawing inspiration from Retentive Networks (RetNet), RetSeg is designed to bridge the gap between precise polyp segmentation and resource utilization, particularly tailored for colonoscopy images. We train and validate RetSeg for polyp segmentation employing two publicly available datasets: Kvasir-SEG and CVC-ClinicDB. Additionally, we showcase RetSeg's promising performance across diverse public datasets, including CVC-ColonDB, ETIS-LaribPolypDB, CVC-300, and BKAI-IGH NeoPolyp. While our work represents an early-stage exploration, further in-depth studies are imperative to advance these promising findings.