# Exploiting the potential of unlabeled endoscopic video data with self-supervised learning

arXiv: https://arxiv.org/abs/1711.09726 · HTML: https://mlsanity.com/paper/1711.09726/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2017.

## Built on by (most cited first, 30 of 33)

- [Self-supervised learning methods and applications in medical imaging analysis: A survey](https://mlsanity.com/paper/2109.08685/) (2021)
- [Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learning](https://mlsanity.com/paper/2102.10680/) (2021)
- [Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures](https://mlsanity.com/paper/2307.15220/) (2023)
- [Dissecting Self-Supervised Learning Methods for Surgical Computer Vision](https://mlsanity.com/paper/2207.00449/) (2022)
- [ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics](https://mlsanity.com/paper/2111.13424/) (2021)
- [OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding](https://mlsanity.com/paper/2406.07471/) (2024)
- [Image Compositing for Segmentation of Surgical Tools without Manual Annotations](https://mlsanity.com/paper/2102.09528/) (2021)
- [Self-Supervised Learning for Endoscopic Video Analysis](https://mlsanity.com/paper/2308.12394/) (2023)
- [FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation](https://mlsanity.com/paper/2202.08141/) (2022)
- [A novel multiple instance learning framework for COVID-19 severity assessment via data augmentation and self-supervised learning](https://mlsanity.com/paper/2102.03837/) (2021)
- [Surgical Visual Domain Adaptation: Results from the MICCAI 2020 SurgVisDom Challenge](https://mlsanity.com/paper/2102.13644/) (2021)
- [Deep Learning for Surgical Instrument Recognition and Segmentation in Robotic-Assisted Surgeries: A Systematic Review](https://mlsanity.com/paper/2410.07269/) (2024)
- [A Kinematic Bottleneck Approach For Pose Regression of Flexible Surgical Instruments directly from Images](https://mlsanity.com/paper/2103.00586/) (2021)
- [Simulation-to-Real domain adaptation with teacher-student learning for endoscopic instrument segmentation](https://mlsanity.com/paper/2103.01593/) (2021)
- [Co-Generation and Segmentation for Generalized Surgical Instrument Segmentation on Unlabelled Data](https://mlsanity.com/paper/2103.09276/) (2021)
- [Jumpstarting Surgical Computer Vision](https://mlsanity.com/paper/2312.05968/) (2023)
- [Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review](https://mlsanity.com/paper/2502.14886/) (2025)
- [MISSU: 3D Medical Image Segmentation via Self-distilling TransUNet](https://mlsanity.com/paper/2206.00902/) (2022)
- [Task Fingerprinting for Meta Learning in Biomedical Image Analysis](https://mlsanity.com/paper/2107.03949/) (2021)
- [Reducing Annotating Load: Active Learning with Synthetic Images in Surgical Instrument Segmentation](https://mlsanity.com/paper/2108.03534/) (2021)
- [Data efficient deep learning for medical image analysis: A survey](https://mlsanity.com/paper/2310.06557/) (2023)
- [Learning of Inter-Label Geometric Relationships Using Self-Supervised Learning: Application To Gleason Grade Segmentation](https://mlsanity.com/paper/2110.00404/) (2021)
- [Self-supervised learning via inter-modal reconstruction and feature projection networks for label-efficient 3D-to-2D segmentation](https://mlsanity.com/paper/2307.03008/) (2023)
- [SegMatch: A semi-supervised learning method for surgical instrument segmentation](https://mlsanity.com/paper/2308.05232/) (2023)
- [A comprehensive survey on recent deep learning-based methods applied to surgical data](https://mlsanity.com/paper/2209.01435/) (2022)
- [SAF-IS: a Spatial Annotation Free Framework for Instance Segmentation of Surgical Tools](https://mlsanity.com/paper/2309.01723/) (2023)
- [Predicting the Timing of Camera Movements From the Kinematics of Instruments in Robotic-Assisted Surgery Using Artificial Neural Networks](https://mlsanity.com/paper/2109.11192/) (2021)
- [DrasCLR: A Self-supervised Framework of Learning Disease-related and Anatomy-specific Representation for 3D Medical Images](https://mlsanity.com/paper/2302.10390/) (2023)
- [Multi-modal Masked Siamese Network Improves Chest X-Ray Representation Learning](https://mlsanity.com/paper/2407.04449/) (2024)
- [Self-supervised Learning via Cluster Distance Prediction for Operating Room Context Awareness](https://mlsanity.com/paper/2407.05448/) (2024)

## Abstract (arXiv)

Surgical data science is a new research field that aims to observe all aspects of the patient treatment process in order to provide the right assistance at the right time. Due to the breakthrough successes of deep learning-based solutions for automatic image annotation, the availability of reference annotations for algorithm training is becoming a major bottleneck in the field. The purpose of this paper was to investigate the concept of self-supervised learning to address this issue. Our approach is guided by the hypothesis that unlabeled video data can be used to learn a representation of the target domain that boosts the performance of state-of-the-art machine learning algorithms when used for pre-training. Core of the method is an auxiliary task based on raw endoscopic video data of the target domain that is used to initialize the convolutional neural network (CNN) for the target task. In this paper, we propose the re-colorization of medical images with a generative adversarial network (GAN)-based architecture as auxiliary task. A variant of the method involves a second pre-training step based on labeled data for the target task from a related domain. We validate both variants using medical instrument segmentation as target task. The proposed approach can be used to radically reduce the manual annotation effort involved in training CNNs. Compared to the baseline approach of generating annotated data from scratch, our method decreases exploratively the number of labeled images by up to 75% without sacrificing performance. Our method also outperforms alternative methods for CNN pre-training, such as pre-training on publicly available non-medical or medical data using the target task (in this instance: segmentation). As it makes efficient use of available (non-)public and (un-)labeled data, the approach has the potential to become a valuable tool for CNN (pre-)training.