# Contrastive Transformer-based Multiple Instance Learning for Weakly Supervised Polyp Frame Detection

arXiv: https://arxiv.org/abs/2203.12121 · HTML: https://mlsanity.com/paper/2203.12121/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 29 references that are in the map.
- First posted on arXiv 23 Mar 2022.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision](https://mlsanity.com/paper/2007.04687/) (2020)
- [Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning](https://mlsanity.com/paper/2101.10030/) (2021)
- [Graph Convolutional Label Noise Cleaner: Train a Plug-and-play Action Classifier for Anomaly Detection](https://mlsanity.com/paper/1903.07256/) (2019)
- [Deep Anomaly Detection with Deviation Networks](https://mlsanity.com/paper/1911.08623/) (2019)
- [MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection](https://mlsanity.com/paper/2104.01633/) (2021)
- [CLAWS: Clustering Assisted Weakly Supervised Learning with Normalcy Suppression for Anomalous Event Detection](https://mlsanity.com/paper/2011.12077/) (2020)
- [Progressively Normalized Self-Attention Network for Video Polyp Segmentation](https://mlsanity.com/paper/2105.08468/) (2021)
- [Deep One-Class Classification via Interpolated Gaussian Descriptor](https://mlsanity.com/paper/2101.10043/) (2021)
- [Constrained Contrastive Distribution Learning for Unsupervised Anomaly Detection and Localisation in Medical Images](https://mlsanity.com/paper/2103.03423/) (2021)
- [Weakly Supervised Video Anomaly Detection via Center-guided Discriminative Learning](https://mlsanity.com/paper/2104.07268/) (2021)
- [Toward Deep Supervised Anomaly Detection: Reinforcement Learning from Partially Labeled Anomaly Data](https://mlsanity.com/paper/2009.06847/) (2020)
- [Few-Shot Anomaly Detection for Polyp Frames from Colonoscopy](https://mlsanity.com/paper/2006.14811/) (2020)
- [Unsupervised Dual Adversarial Learning for Anomaly Detection in Colonoscopy Video Frames](https://mlsanity.com/paper/1910.10345/) (2019)
- [Detecting, Localising and Classifying Polyps from Colonoscopy Videos using Deep Learning](https://mlsanity.com/paper/2101.03285/) (2021)

## Built on by (most cited first, 20 of 20)

- [Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train](https://mlsanity.com/paper/2306.16741/) (2023)
- [Endora: Video Generation Models as Endoscopy Simulators](https://mlsanity.com/paper/2403.11050/) (2024)
- [Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives](https://mlsanity.com/paper/2206.01136/) (2022)
- [OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding](https://mlsanity.com/paper/2406.07471/) (2024)
- [FairDomain: Achieving Fairness in Cross-Domain Medical Image Segmentation and Classification](https://mlsanity.com/paper/2407.08813/) (2024)
- [MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation](https://mlsanity.com/paper/2409.19684/) (2024)
- [Frontiers in Intelligent Colonoscopy](https://mlsanity.com/paper/2410.17241/) (2024)
- [EndoMamba: An Efficient Foundation Model for Endoscopic Videos via Hierarchical Pre-training](https://mlsanity.com/paper/2502.19090/) (2025)
- [Optical Flow Representation Alignment Mamba Diffusion Model for Medical Video Generation](https://mlsanity.com/paper/2411.01647/) (2024)
- [Towards Hierarchical Regional Transformer-based Multiple Instance Learning](https://mlsanity.com/paper/2308.12634/) (2023)
- [Cross- and Intra-image Prototypical Learning for Multi-label Disease Diagnosis and Interpretation](https://mlsanity.com/paper/2411.04607/) (2024)
- [FEAT: Full-Dimensional Efficient Attention Transformer for Medical Video Generation](https://mlsanity.com/paper/2506.04956/) (2025)
- [SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos](https://mlsanity.com/paper/2602.05638/) (2026)
- [Arges: Spatio-Temporal Transformer for Ulcerative Colitis Severity Assessment in Endoscopy Videos](https://mlsanity.com/paper/2410.00536/) (2024)
- [Surgical Foundation Model Leveraging Compression and Entropy Maximization for Image-Guided Surgical Assistance](https://mlsanity.com/paper/2506.01980/) (2025)
- [MEGAN: Mixture of Experts for Robust Uncertainty Estimation in Endoscopy Videos](https://mlsanity.com/paper/2509.12772/) (2025)
- [Token Merging via Spatiotemporal Information Mining for Surgical Video Understanding](https://mlsanity.com/paper/2509.23672/) (2025)
- [Focus-to-Perceive Representation Learning: A Cognition-Inspired Hierarchical Framework for Endoscopic Video Analysis](https://mlsanity.com/paper/2603.25778/) (2026)
- [Clinically-Grounded Counterfactual Reasoning for Medical Video Diagnosis](https://mlsanity.com/paper/2605.26483/) (2026)
- [EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment](https://mlsanity.com/paper/2608.04472/) (2026)

## Abstract (arXiv)

Current polyp detection methods from colonoscopy videos use exclusively normal (i.e., healthy) training images, which i) ignore the importance of temporal information in consecutive video frames, and ii) lack knowledge about the polyps. Consequently, they often have high detection errors, especially on challenging polyp cases (e.g., small, flat, or partially visible polyps). In this work, we formulate polyp detection as a weakly-supervised anomaly detection task that uses video-level labelled training data to detect frame-level polyps. In particular, we propose a novel convolutional transformer-based multiple instance learning method designed to identify abnormal frames (i.e., frames with polyps) from anomalous videos (i.e., videos containing at least one frame with polyp). In our method, local and global temporal dependencies are seamlessly captured while we simultaneously optimise video and snippet-level anomaly scores. A contrastive snippet mining method is also proposed to enable an effective modelling of the challenging polyp cases. The resulting method achieves a detection accuracy that is substantially better than current state-of-the-art approaches on a new large-scale colonoscopy video dataset introduced in this work.