# Dissecting Self-Supervised Learning Methods for Surgical Computer Vision

arXiv: https://arxiv.org/abs/2207.00449 · HTML: https://mlsanity.com/paper/2207.00449/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 95 of its 126 references that are in the map.
- First posted on arXiv 1 Jul 2022.

## Builds on (most cited first, 30 of 95)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)

## Built on by (most cited first, 30 of 48)

- [Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures](https://mlsanity.com/paper/2307.15220/) (2023)
- [Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training](https://mlsanity.com/paper/2305.13689/) (2023)
- [Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation](https://mlsanity.com/paper/2410.00263/) (2024)
- [Latent Graph Representations for Critical View of Safety Assessment](https://mlsanity.com/paper/2212.04155/) (2022)
- [Scaling up self-supervised learning for improved surgical foundation models](https://mlsanity.com/paper/2501.09436/) (2025)
- [Self-Supervised Learning for Endoscopic Video Analysis](https://mlsanity.com/paper/2308.12394/) (2023)
- [The Endoscapes Dataset for Surgical Scene Segmentation, Object Detection, and Critical View of Safety Assessment: Official Splits and Benchmark](https://mlsanity.com/paper/2312.12429/) (2023)
- [LEMON: A Large Endoscopic MONocular Dataset and Foundation Model for Perception in Surgical Settings](https://mlsanity.com/paper/2503.19740/) (2025)
- [CholecTrack20: A Multi-Perspective Tracking Dataset for Surgical Tools](https://mlsanity.com/paper/2312.07352/) (2023)
- [Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoning](https://mlsanity.com/paper/2312.06829/) (2023)
- [Large-scale Self-supervised Video Foundation Model for Intelligent Surgery](https://mlsanity.com/paper/2506.02692/) (2025)
- [Jumpstarting Surgical Computer Vision](https://mlsanity.com/paper/2312.05968/) (2023)
- [Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review](https://mlsanity.com/paper/2502.14886/) (2025)
- [SSL-CPCD: Self-supervised learning with composite pretext-class discrimination for improved generalisability in endoscopic image analysis](https://mlsanity.com/paper/2306.00197/) (2023)
- [Exploring the Effect of Dataset Diversity in Self-Supervised Learning for Surgical Computer Vision](https://mlsanity.com/paper/2407.17904/) (2024)
- [TUNeS: A Temporal U-Net with Self-Attention for Video-based Surgical Phase Recognition](https://mlsanity.com/paper/2307.09997/) (2023)
- [Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligence](https://mlsanity.com/paper/2504.02799/) (2025)
- [Swin transformers are robust to distribution and concept drift in endoscopy-based longitudinal rectal cancer assessment](https://mlsanity.com/paper/2405.03762/) (2024)
- [Surgical Triplet Recognition via Diffusion Model](https://mlsanity.com/paper/2406.13210/) (2024)
- [Recognizing Surgical Phases Anywhere: Few-Shot Test-time Adaptation and Task-graph Guided Refinement](https://mlsanity.com/paper/2506.20254/) (2025)
- [SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos](https://mlsanity.com/paper/2602.05638/) (2026)
- [DDA: Dimensionality Driven Augmentation Search for Contrastive Learning in Laparoscopic Surgery](https://mlsanity.com/paper/2406.00907/) (2024)
- [CycleSAM: Few-Shot Surgical Scene Segmentation with Cycle- and Scene-Consistent Feature Matching](https://mlsanity.com/paper/2407.06795/) (2024)
- [SurgPETL: Parameter-Efficient Image-to-Surgical-Video Transfer Learning for Surgical Phase Recognition](https://mlsanity.com/paper/2409.20083/) (2024)
- [Federated EndoViT: Pretraining Vision Transformers via Federated Learning on Endoscopic Image Collections](https://mlsanity.com/paper/2504.16612/) (2025)
- [Cataract-LMM Large-Scale Multi-Source Multi-Task Benchmark for Deep Learning in Surgical Video Analysis](https://mlsanity.com/paper/2510.16371/) (2025)
- [SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understanding](https://mlsanity.com/paper/2511.21339/) (2025)
- [SAF-IS: a Spatial Annotation Free Framework for Instance Segmentation of Surgical Tools](https://mlsanity.com/paper/2309.01723/) (2023)
- [A Study on Self-Supervised Pretraining for Vision Problems in Gastrointestinal Endoscopy](https://mlsanity.com/paper/2401.06278/) (2024)
- [Exploring Pre-training Across Domains for Few-Shot Surgical Skill Assessment](https://mlsanity.com/paper/2509.09327/) (2025)

## Abstract (arXiv)

The field of surgical computer vision has undergone considerable breakthroughs in recent years with the rising popularity of deep neural network-based methods. However, standard fully-supervised approaches for training such models require vast amounts of annotated data, imposing a prohibitively high cost; especially in the clinical domain. Self-Supervised Learning (SSL) methods, which have begun to gain traction in the general computer vision community, represent a potential solution to these annotation costs, allowing to learn useful representations from only unlabeled data. Still, the effectiveness of SSL methods in more complex and impactful domains, such as medicine and surgery, remains limited and unexplored. In this work, we address this critical need by investigating four state-of-the-art SSL methods (MoCo v2, SimCLR, DINO, SwAV) in the context of surgical computer vision. We present an extensive analysis of the performance of these methods on the Cholec80 dataset for two fundamental and popular tasks in surgical context understanding, phase recognition and tool presence detection. We examine their parameterization, then their behavior with respect to training data quantities in semi-supervised settings. Correct transfer of these methods to surgery, as described and conducted in this work, leads to substantial performance gains over generic uses of SSL - up to 7.4% on phase recognition and 20% on tool presence detection - as well as state-of-the-art semi-supervised phase recognition approaches by up to 14%. Further results obtained on a highly diverse selection of surgical datasets exhibit strong generalization properties. The code is available at https://github.com/CAMMA-public/SelfSupSurg.