# Self-Supervised Learning for Endoscopic Video Analysis

arXiv: https://arxiv.org/abs/2308.12394 · HTML: https://mlsanity.com/paper/2308.12394/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 23 Aug 2023.

## Builds on (most cited first, 23 of 23)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [Masked Siamese Networks for Label-Efficient Learning](https://mlsanity.com/paper/2204.07141/) (2022)
- [TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks](https://mlsanity.com/paper/2003.10751/) (2020)
- [SCENIC: A JAX Library for Computer Vision Research and Beyond](https://mlsanity.com/paper/2110.11403/) (2021)
- [Regularization by Denoising via Fixed-Point Projection (RED-PRO)](https://mlsanity.com/paper/2008.00226/) (2020)
- [Dissecting Self-Supervised Learning Methods for Surgical Computer Vision](https://mlsanity.com/paper/2207.00449/) (2022)
- [Colonoscopy Polyp Detection and Classification: Dataset Creation and Comparative Evaluations](https://mlsanity.com/paper/2104.10824/) (2021)
- [Exploiting the potential of unlabeled endoscopic video data with self-supervised learning](https://mlsanity.com/paper/1711.09726/) (2017)
- [Self-Supervised Surgical Tool Segmentation using Kinematic Information](https://mlsanity.com/paper/1902.04810/) (2019)
- [A Kinematic Bottleneck Approach For Pose Regression of Flexible Surgical Instruments directly from Images](https://mlsanity.com/paper/2103.00586/) (2021)
- [Self-Supervised Polyp Re-Identification in Colonoscopy](https://mlsanity.com/paper/2306.08591/) (2023)
- [Bounded Future MS-TCN++ for surgical gesture recognition](https://mlsanity.com/paper/2209.14647/) (2022)

## Built on by (most cited first, 22 of 22)

- [Scaling up self-supervised learning for improved surgical foundation models](https://mlsanity.com/paper/2501.09436/) (2025)
- [LEMON: A Large Endoscopic MONocular Dataset and Foundation Model for Perception in Surgical Settings](https://mlsanity.com/paper/2503.19740/) (2025)
- [OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understanding](https://mlsanity.com/paper/2504.14692/) (2025)
- [Large-scale Self-supervised Video Foundation Model for Intelligent Surgery](https://mlsanity.com/paper/2506.02692/) (2025)
- [Jumpstarting Surgical Computer Vision](https://mlsanity.com/paper/2312.05968/) (2023)
- [Exploring the Effect of Dataset Diversity in Self-Supervised Learning for Surgical Computer Vision](https://mlsanity.com/paper/2407.17904/) (2024)
- [SciVid: Cross-Domain Evaluation of Video Models in Scientific Applications](https://mlsanity.com/paper/2507.03578/) (2025)
- [Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligence](https://mlsanity.com/paper/2504.02799/) (2025)
- [SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos](https://mlsanity.com/paper/2602.05638/) (2026)
- [SurgPETL: Parameter-Efficient Image-to-Surgical-Video Transfer Learning for Surgical Phase Recognition](https://mlsanity.com/paper/2409.20083/) (2024)
- [A Study on Self-Supervised Pretraining for Vision Problems in Gastrointestinal Endoscopy](https://mlsanity.com/paper/2401.06278/) (2024)
- [EndoFinder: Online Image Retrieval for Explainable Colorectal Polyp Diagnosis](https://mlsanity.com/paper/2407.11401/) (2024)
- [Arges: Spatio-Temporal Transformer for Ulcerative Colitis Severity Assessment in Endoscopy Videos](https://mlsanity.com/paper/2410.00536/) (2024)
- [Unsupervised Segmentation of Colonoscopy Images](https://mlsanity.com/paper/2312.12599/) (2023)
- [Predicting Generalization of AI Colonoscopy Models to Unseen Data](https://mlsanity.com/paper/2403.09920/) (2024)
- [Surgical Foundation Model Leveraging Compression and Entropy Maximization for Image-Guided Surgical Assistance](https://mlsanity.com/paper/2506.01980/) (2025)
- [MEGAN: Mixture of Experts for Robust Uncertainty Estimation in Endoscopy Videos](https://mlsanity.com/paper/2509.12772/) (2025)
- [On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-training](https://mlsanity.com/paper/2601.18929/) (2026)
- [Focus-to-Perceive Representation Learning: A Cognition-Inspired Hierarchical Framework for Endoscopic Video Analysis](https://mlsanity.com/paper/2603.25778/) (2026)
- [CapCLIP: A Vision-Language Representation Alignment Approach for Wireless Capsule Endoscopy Analysis](https://mlsanity.com/paper/2605.08493/) (2026)
- [Task-Aligned Self-Supervised Learning for Medical Image Analysis: A Task-Oriented Review with Practical Design Guidelines](https://mlsanity.com/paper/2605.23995/) (2026)
- [DART: Depth-as-Target Pretraining for Surgical Vision Foundation Models](https://mlsanity.com/paper/2609.04555/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) has led to important breakthroughs in computer vision by allowing learning from large amounts of unlabeled data. As such, it might have a pivotal role to play in biomedicine where annotating data requires a highly specialized expertise. Yet, there are many healthcare domains for which SSL has not been extensively explored. One such domain is endoscopy, minimally invasive procedures which are commonly used to detect and treat infections, chronic inflammatory diseases or cancer. In this work, we study the use of a leading SSL framework, namely Masked Siamese Networks (MSNs), for endoscopic video analysis such as colonoscopy and laparoscopy. To fully exploit the power of SSL, we create sizable unlabeled endoscopic video datasets for training MSNs. These strong image representations serve as a foundation for secondary training with limited annotated datasets, resulting in state-of-the-art performance in endoscopic benchmarks like surgical phase recognition during laparoscopy and colonoscopic polyp characterization. Additionally, we achieve a 50% reduction in annotated data size without sacrificing performance. Thus, our work provides evidence that SSL can dramatically reduce the need of annotated data in endoscopy.