# Towards Automated Polyp Segmentation Using Weakly- and Semi-Supervised Learning and Deformable Transformers

arXiv: https://arxiv.org/abs/2211.11847 · HTML: https://mlsanity.com/paper/2211.11847/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Builds on (most cited first, 30 of 31)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [Enhanced-alignment Measure for Binary Foreground Map Evaluation](https://mlsanity.com/paper/1805.10421/) (2018)
- [ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring](https://mlsanity.com/paper/1911.09785/) (2019)
- [Label Propagation for Deep Semi-supervised Learning](https://mlsanity.com/paper/1904.04717/) (2019)
- [Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations](https://mlsanity.com/paper/1904.05044/) (2019)
- [A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images](https://mlsanity.com/paper/1612.00799/) (2016)
- [CE-Net: Context Encoder Network for 2D Medical Image Segmentation](https://mlsanity.com/paper/1903.02740/) (2019)
- [A Simple Pooling-Based Design for Real-Time Salient Object Detection](https://mlsanity.com/paper/1904.09569/) (2019)
- [ResUNet++: An Advanced Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1911.07067/) (2019)
- [ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning](https://mlsanity.com/paper/2007.07936/) (2020)
- [Normalized Cut Loss for Weakly-supervised CNN Segmentation](https://mlsanity.com/paper/1804.01346/) (2018)
- [Weakly-Supervised Salient Object Detection via Scribble Annotations](https://mlsanity.com/paper/2003.07685/) (2020)
- [InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting](https://mlsanity.com/paper/1908.07801/) (2019)
- [A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation](https://mlsanity.com/paper/2104.07256/) (2021)
- [Structure-Consistent Weakly Supervised Salient Object Detection with Local Saliency Coherence](https://mlsanity.com/paper/2012.04404/) (2020)
- [Learning to Segment via Cut-and-Paste](https://mlsanity.com/paper/1803.06414/) (2018)
- [NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentation](https://mlsanity.com/paper/2208.14876/) (2022)
- [Patcher: Patch Transformers with Mixture of Experts for Precise Medical Image Segmentation](https://mlsanity.com/paper/2206.01741/) (2022)

## Built on by (most cited first, 6 of 6)

- [Frontiers in Intelligent Colonoscopy](https://mlsanity.com/paper/2410.17241/) (2024)
- [Knowledge Extraction and Distillation from Large-Scale Image-Text Colonoscopy Records Leveraging Large Language and Vision Models](https://mlsanity.com/paper/2310.11173/) (2023)
- [SCALER: SAM-Enhanced Collaborative Learning for Label-Deficient Concealed Object Segmentation](https://mlsanity.com/paper/2511.18136/) (2025)
- [Rethinking Polyp Segmentation from an Out-of-Distribution Perspective](https://mlsanity.com/paper/2306.07792/) (2023)
- [Segment Concealed Objects with Incomplete Supervision](https://mlsanity.com/paper/2506.08955/) (2025)
- [ToNNO: Tomographic Reconstruction of a Neural Network's Output for Weakly Supervised Segmentation of 3D Medical Images](https://mlsanity.com/paper/2404.13103/) (2024)

## Abstract (arXiv)

Polyp segmentation is a crucial step towards computer-aided diagnosis of colorectal cancer. However, most of the polyp segmentation methods require pixel-wise annotated datasets. Annotated datasets are tedious and time-consuming to produce, especially for physicians who must dedicate their time to their patients. We tackle this issue by proposing a novel framework that can be trained using only weakly annotated images along with exploiting unlabeled images. To this end, we propose three ideas to address this problem, more specifically our contributions are: 1) a novel sparse foreground loss that suppresses false positives and improves weakly-supervised training, 2) a batch-wise weighted consistency loss utilizing predicted segmentation maps from identical networks trained using different initialization during semi-supervised training, 3) a deformable transformer encoder neck for feature enhancement by fusing information across levels and flexible spatial locations. Extensive experimental results demonstrate the merits of our ideas on five challenging datasets outperforming some state-of-the-art fully supervised models. Also, our framework can be utilized to fine-tune models trained on natural image segmentation datasets drastically improving their performance for polyp segmentation and impressively demonstrating superior performance to fully supervised fine-tuning.