# Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers

arXiv: https://arxiv.org/abs/2108.06932 · HTML: https://mlsanity.com/paper/2108.06932/

## Key facts

- Cited by 159 papers in the map (data through 14 Oct 2026).
- Builds on 73 of its 101 references that are in the map.
- First posted on arXiv 16 Aug 2021.

## Builds on (most cited first, 30 of 73)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [Structure-measure: A New Way to Evaluate Foreground Maps](https://mlsanity.com/paper/1708.00786/) (2017)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [Enhanced-alignment Measure for Binary Foreground Map Evaluation](https://mlsanity.com/paper/1805.10421/) (2018)
- [Rethinking Spatial Dimensions of Vision Transformers](https://mlsanity.com/paper/2103.16302/) (2021)
- [TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation](https://mlsanity.com/paper/2102.08005/) (2021)
- [Understanding Robustness of Transformers for Image Classification](https://mlsanity.com/paper/2103.14586/) (2021)

## Built on by (most cited first, 30 of 159)

- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [Video Polyp Segmentation: A Deep Learning Perspective](https://mlsanity.com/paper/2203.14291/) (2022)
- [High-resolution Iterative Feedback Network for Camouflaged Object Detection](https://mlsanity.com/paper/2203.11624/) (2022)
- [AutoSAM: Adapting SAM to Medical Images by Overloading the Prompt Encoder](https://mlsanity.com/paper/2306.06370/) (2023)
- [Polyp-SAM: Transfer SAM for Polyp Segmentation](https://mlsanity.com/paper/2305.00293/) (2023)
- [EMCAD: Efficient Multi-scale Convolutional Attention Decoding for Medical Image Segmentation](https://mlsanity.com/paper/2405.06880/) (2024)
- [Can SAM Segment Polyps?](https://mlsanity.com/paper/2304.07583/) (2023)
- [Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation](https://mlsanity.com/paper/2304.12637/) (2023)
- [A Survey of Camouflaged Object Detection and Beyond](https://mlsanity.com/paper/2408.14562/) (2024)
- [Multi-scale Hierarchical Vision Transformer with Cascaded Attention Decoding for Medical Image Segmentation](https://mlsanity.com/paper/2303.16892/) (2023)
- [Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives](https://mlsanity.com/paper/2206.01136/) (2022)
- [ArSDM: Colonoscopy Images Synthesis with Adaptive Refinement Semantic Diffusion Models](https://mlsanity.com/paper/2309.01111/) (2023)
- [Multi-Scale and Detail-Enhanced Segment Anything Model for Salient Object Detection](https://mlsanity.com/paper/2408.04326/) (2024)
- [Personalizing Federated Medical Image Segmentation via Local Calibration](https://mlsanity.com/paper/2207.04655/) (2022)
- [RUN: Reversible Unfolding Network for Concealed Object Segmentation](https://mlsanity.com/paper/2501.18783/) (2025)
- [MedSAM3: Delving into Segment Anything with Medical Concepts](https://mlsanity.com/paper/2511.19046/) (2025)
- [Boosting Camouflaged Object Detection with Dual-Task Interactive Transformer](https://mlsanity.com/paper/2205.10579/) (2022)
- [U-Net v2: Rethinking the Skip Connections of U-Net for Medical Image Segmentation](https://mlsanity.com/paper/2311.17791/) (2023)
- [Noise-Consistent Siamese-Diffusion for Medical Image Synthesis and Segmentation](https://mlsanity.com/paper/2505.06068/) (2025)
- [DuAT: Dual-Aggregation Transformer Network for Medical Image Segmentation](https://mlsanity.com/paper/2212.11677/) (2022)
- [Polyp-SAM++: Can A Text Guided SAM Perform Better for Polyp Segmentation?](https://mlsanity.com/paper/2308.06623/) (2023)
- [CamoDiffusion: Camouflaged Object Detection via Conditional Diffusion Models](https://mlsanity.com/paper/2305.17932/) (2023)
- [WeakPolyp: You Only Look Bounding Box for Polyp Segmentation](https://mlsanity.com/paper/2307.10912/) (2023)
- [G-CASCADE: Efficient Cascaded Graph Convolutional Decoding for 2D Medical Image Segmentation](https://mlsanity.com/paper/2310.16175/) (2023)
- [SALI: Short-term Alignment and Long-term Interaction Network for Colonoscopy Video Polyp Segmentation](https://mlsanity.com/paper/2406.13532/) (2024)
- [CamDiff: Camouflage Image Augmentation via Diffusion Model](https://mlsanity.com/paper/2304.05469/) (2023)
- [Dual Cross-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2303.17696/) (2023)
- [Denoising Diffusion Semantic Segmentation with Mask Prior Modeling](https://mlsanity.com/paper/2306.01721/) (2023)
- [1M parameters are enough? A lightweight CNN-based model for medical image segmentation](https://mlsanity.com/paper/2306.16103/) (2023)
- [MH-pFLID: Model Heterogeneous personalized Federated Learning via Injection and Distillation for Medical Data Analysis](https://mlsanity.com/paper/2405.06822/) (2024)

## Abstract (arXiv)

Most polyp segmentation methods use CNNs as their backbone, leading to two key issues when exchanging information between the encoder and decoder: 1) taking into account the differences in contribution between different-level features and 2) designing an effective mechanism for fusing these features. Unlike existing CNN-based methods, we adopt a transformer encoder, which learns more powerful and robust representations. In addition, considering the image acquisition influence and elusive properties of polyps, we introduce three standard modules, including a cascaded fusion module (CFM), a camouflage identification module (CIM), and a similarity aggregation module (SAM). Among these, the CFM is used to collect the semantic and location information of polyps from high-level features; the CIM is applied to capture polyp information disguised in low-level features, and the SAM extends the pixel features of the polyp area with high-level semantic position information to the entire polyp area, thereby effectively fusing cross-level features. The proposed model, named Polyp-PVT, effectively suppresses noises in the features and significantly improves their expressive capabilities. Extensive experiments on five widely adopted datasets show that the proposed model is more robust to various challenging situations (e.g., appearance changes, small objects, rotation) than existing representative methods. The proposed model is available at https://github.com/DengPingFan/Polyp-PVT.