# Detecting, Localising and Classifying Polyps from Colonoscopy Videos using Deep Learning

arXiv: https://arxiv.org/abs/2101.03285 · HTML: https://mlsanity.com/paper/2101.03285/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 63 references that are in the map.
- First posted on arXiv 9 Jan 2021.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [Adversarial Autoencoders](https://mlsanity.com/paper/1511.05644/) (2015)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representations](https://mlsanity.com/paper/1903.08550/) (2019)
- [Deep Anomaly Detection with Deviation Networks](https://mlsanity.com/paper/1911.08623/) (2019)
- [Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation](https://mlsanity.com/paper/1808.01200/) (2018)
- [Learning Representations of Ultrahigh-dimensional Data for Random Distance-based Outlier Detection](https://mlsanity.com/paper/1806.04808/) (2018)
- [Overfitting of neural nets under class imbalance: Analysis and improvements for segmentation](https://mlsanity.com/paper/1907.10982/) (2019)
- [Deep-Learning for Classification of Colorectal Polyps on Whole-Slide Images](https://mlsanity.com/paper/1703.01550/) (2017)
- [Towards safe deep learning: accurately quantifying biomarker uncertainty in neural network predictions](https://mlsanity.com/paper/1806.08640/) (2018)
- [Few-Shot Anomaly Detection for Polyp Frames from Colonoscopy](https://mlsanity.com/paper/2006.14811/) (2020)
- [Unsupervised Dual Adversarial Learning for Anomaly Detection in Colonoscopy Video Frames](https://mlsanity.com/paper/1910.10345/) (2019)
- [XNet: A convolutional neural network (CNN) implementation for medical X-Ray image segmentation suitable for small datasets](https://mlsanity.com/paper/1812.00548/) (2018)

## Built on by (most cited first, 3 of 3)

- [Constrained Contrastive Distribution Learning for Unsupervised Anomaly Detection and Localisation in Medical Images](https://mlsanity.com/paper/2103.03423/) (2021)
- [Self-supervised Pseudo Multi-class Pre-training for Unsupervised Anomaly Detection and Segmentation in Medical Images](https://mlsanity.com/paper/2109.01303/) (2021)
- [Contrastive Transformer-based Multiple Instance Learning for Weakly Supervised Polyp Frame Detection](https://mlsanity.com/paper/2203.12121/) (2022)

## Abstract (arXiv)

In this paper, we propose and analyse a system that can automatically detect, localise and classify polyps from colonoscopy videos. The detection of frames with polyps is formulated as a few-shot anomaly classification problem, where the training set is highly imbalanced with the large majority of frames consisting of normal images and a small minority comprising frames with polyps. Colonoscopy videos may contain blurry images and frames displaying feces and water jet sprays to clean the colon -- such frames can mistakenly be detected as anomalies, so we have implemented a classifier to reject these two types of frames before polyp detection takes place. Next, given a frame containing a polyp, our method localises (with a bounding box around the polyp) and classifies it into five different classes. Furthermore, we study a method to improve the reliability and interpretability of the classification result using uncertainty estimation and classification calibration. Classification uncertainty and calibration not only help improve classification accuracy by rejecting low-confidence and high-uncertain results, but can be used by doctors to decide how to decide on the classification of a polyp. All the proposed detection, localisation and classification methods are tested using large data sets and compared with relevant baseline approaches.