# WeakPolyp: You Only Look Bounding Box for Polyp Segmentation

arXiv: https://arxiv.org/abs/2307.10912 · HTML: https://mlsanity.com/paper/2307.10912/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 23 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 15 of 15)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation](https://mlsanity.com/paper/2102.08005/) (2021)
- [Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers](https://mlsanity.com/paper/2108.06932/) (2021)
- [Shallow Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2108.00882/) (2021)
- [Video Polyp Segmentation: A Deep Learning Perspective](https://mlsanity.com/paper/2203.14291/) (2022)
- [Automatic Polyp Segmentation via Multi-scale Subtraction Network](https://mlsanity.com/paper/2108.05082/) (2021)
- [Progressively Normalized Self-Attention Network for Video Polyp Segmentation](https://mlsanity.com/paper/2105.08468/) (2021)
- [TGANet: Text-guided attention for improved polyp segmentation](https://mlsanity.com/paper/2205.04280/) (2022)
- [Stepwise Feature Fusion: Local Guides Global](https://mlsanity.com/paper/2203.03635/) (2022)
- [Adaptive Context Selection for Polyp Segmentation](https://mlsanity.com/paper/2301.04799/) (2023)
- [Lesion-aware Dynamic Kernel for Polyp Segmentation](https://mlsanity.com/paper/2301.04904/) (2023)
- [BoxPolyp:Boost Generalized Polyp Segmentation Using Extra Coarse Bounding Box Annotations](https://mlsanity.com/paper/2212.03498/) (2022)

## Built on by (most cited first, 21 of 21)

- [Spider: A Unified Framework for Context-dependent Concept Segmentation](https://mlsanity.com/paper/2405.01002/) (2024)
- [Frontiers in Intelligent Colonoscopy](https://mlsanity.com/paper/2410.17241/) (2024)
- [Inspiring the Next Generation of Segment Anything Models: Comprehensively Evaluate SAM and SAM 2 with Diverse Prompts Towards Context-Dependent Concepts under Different Scenes](https://mlsanity.com/paper/2412.01240/) (2024)
- [TP-Seg: Task-Prototype Framework for Unified Medical Lesion Segmentation](https://mlsanity.com/paper/2604.00684/) (2026)
- [SimTxtSeg: Weakly-Supervised Medical Image Segmentation with Simple Text Cues](https://mlsanity.com/paper/2406.19364/) (2024)
- [Colorectal Polyp Segmentation in the Deep Learning Era: A Comprehensive Survey](https://mlsanity.com/paper/2401.11734/) (2024)
- [MonoBox: Tightness-free Box-supervised Polyp Segmentation using Monotonicity Constraint](https://mlsanity.com/paper/2404.01188/) (2024)
- [From Few to More: Scribble-based Medical Image Segmentation via Masked Context Modeling and Continuous Pseudo Labels](https://mlsanity.com/paper/2408.12814/) (2024)
- [IBoxCLA: Towards Robust Box-supervised Segmentation of Polyp via Improved Box-dice and Contrastive Latent-anchors](https://mlsanity.com/paper/2310.07248/) (2023)
- [HilbertMamba: Local-Global Reciprocal Network for Uterine Fibroid Segmentation in Ultrasound Videos](https://mlsanity.com/paper/2407.05703/) (2024)
- [Rethinking Barely-Supervised Volumetric Medical Image Segmentation from an Unsupervised Domain Adaptation Perspective](https://mlsanity.com/paper/2405.09777/) (2024)
- [MARIO: A Mixed Annotation Framework For Polyp Segmentation](https://mlsanity.com/paper/2501.10957/) (2025)
- [DEPT: Deep Extreme Point Tracing for Ultrasound Image Segmentation](https://mlsanity.com/paper/2503.15260/) (2025)
- [BiSeg-SAM: Weakly-Supervised Post-Processing Framework for Boosting Binary Segmentation in Segment Anything Models](https://mlsanity.com/paper/2504.01452/) (2025)
- [Label tree semantic losses for rich multi-class medical image segmentation](https://mlsanity.com/paper/2507.15777/) (2025)
- [RefineSeg: Dual Coarse-to-Fine Learning for Medical Image Segmentation](https://mlsanity.com/paper/2508.02844/) (2025)
- [SynMatch: Rethinking Consistency in Medical Image Segmentation with Sparse Annotations](https://mlsanity.com/paper/2508.07298/) (2025)
- [Uncertainty-Aware Extreme Point Tracing for Weakly Supervised Ultrasound Image Segmentation](https://mlsanity.com/paper/2510.15666/) (2025)
- [ZScribbleSeg: A comprehensive segmentation framework with modeling of efficient annotation and maximization of scribble supervision](https://mlsanity.com/paper/2605.06266/) (2026)
- [OBBSeg: Irregular Lesion Segmentation under Oriented Bounding Box Annotations](https://mlsanity.com/paper/2607.06007/) (2026)
- [Prompt-Conditioned Channel Attention for Hierarchical Feature Modulation toward Anatomy-Agnostic Segmentation](https://mlsanity.com/paper/2608.20229/) (2026)

## Abstract (arXiv)

Limited by expensive pixel-level labels, polyp segmentation models are plagued by data shortage and suffer from impaired generalization. In contrast, polyp bounding box annotations are much cheaper and more accessible. Thus, to reduce labeling cost, we propose to learn a weakly supervised polyp segmentation model (i.e., WeakPolyp) completely based on bounding box annotations. However, coarse bounding boxes contain too much noise. To avoid interference, we introduce the mask-to-box (M2B) transformation. By supervising the outer box mask of the prediction instead of the prediction itself, M2B greatly mitigates the mismatch between the coarse label and the precise prediction. But, M2B only provides sparse supervision, leading to non-unique predictions. Therefore, we further propose a scale consistency (SC) loss for dense supervision. By explicitly aligning predictions across the same image at different scales, the SC loss largely reduces the variation of predictions. Note that our WeakPolyp is a plug-and-play model, which can be easily ported to other appealing backbones. Besides, the proposed modules are only used during training, bringing no computation cost to inference. Extensive experiments demonstrate the effectiveness of our proposed WeakPolyp, which surprisingly achieves a comparable performance with a fully supervised model, requiring no mask annotations at all.