# FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation

arXiv: https://arxiv.org/abs/2202.08141 · HTML: https://mlsanity.com/paper/2202.08141/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 76 references that are in the map.
- First posted on arXiv 16 Feb 2022.

## Builds on (most cited first, 30 of 46)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Uncertainty-aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentation](https://mlsanity.com/paper/1907.07034/) (2019)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)
- [CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise](https://mlsanity.com/paper/1711.07131/) (2017)
- [Deep Learning is Robust to Massive Label Noise](https://mlsanity.com/paper/1705.10694/) (2017)
- [SELF: Learning to Filter Noisy Labels with Self-Ensembling](https://mlsanity.com/paper/1910.01842/) (2019)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [Webly Supervised Learning of Convolutional Networks](https://mlsanity.com/paper/1505.01554/) (2015)
- [Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning](https://mlsanity.com/paper/1803.01207/) (2018)
- [Unsupervised Moving Object Detection via Contextual Information Separation](https://mlsanity.com/paper/1901.03360/) (2019)
- [Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets](https://mlsanity.com/paper/2007.05405/) (2020)
- [CycleGAN, a Master of Steganography](https://mlsanity.com/paper/1712.02950/) (2017)
- [Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video](https://mlsanity.com/paper/1907.07899/) (2019)
- [Making a Case for 3D Convolutions for Object Segmentation in Videos](https://mlsanity.com/paper/2008.11516/) (2020)
- [E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception](https://mlsanity.com/paper/2107.00229/) (2021)
- [Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery](https://mlsanity.com/paper/1805.02475/) (2018)
- [Learning Motion Flows for Semi-supervised Instrument Segmentation from Robotic Surgical Video](https://mlsanity.com/paper/2007.02501/) (2020)

## Built on by (most cited first, 20 of 20)

- [Latent Graph Representations for Critical View of Safety Assessment](https://mlsanity.com/paper/2212.04155/) (2022)
- [Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art](https://mlsanity.com/paper/2304.13014/) (2023)
- [StegoGAN: Leveraging Steganography for Non-Bijective Image-to-Image Translation](https://mlsanity.com/paper/2403.20142/) (2024)
- [Transferring Relative Monocular Depth to Surgical Vision with Temporal Consistency](https://mlsanity.com/paper/2403.06683/) (2024)
- [Deep Learning for Surgical Instrument Recognition and Segmentation in Robotic-Assisted Surgeries: A Systematic Review](https://mlsanity.com/paper/2410.07269/) (2024)
- [ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding](https://mlsanity.com/paper/2407.19435/) (2024)
- [SANGRIA: Surgical Video Scene Graph Optimization for Surgical Workflow Prediction](https://mlsanity.com/paper/2407.20214/) (2024)
- [Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Rooms](https://mlsanity.com/paper/2404.09231/) (2024)
- [Slot-BERT: Self-supervised Object Discovery in Surgical Video](https://mlsanity.com/paper/2501.12477/) (2025)
- [Revisiting Surgical Instrument Segmentation Without Human Intervention: A Graph Partitioning View](https://mlsanity.com/paper/2408.14789/) (2024)
- [AMNCutter: Affinity-Attention-Guided Multi-View Normalized Cutter for Unsupervised Surgical Instrument Segmentation](https://mlsanity.com/paper/2411.03695/) (2024)
- [SAF-IS: a Spatial Annotation Free Framework for Instance Segmentation of Surgical Tools](https://mlsanity.com/paper/2309.01723/) (2023)
- [Strategies to Improve Real-World Applicability of Laparoscopic Anatomy Segmentation Models](https://mlsanity.com/paper/2403.17192/) (2024)
- [Visual-Kinematics Graph Learning for Procedure-agnostic Instrument Tip Segmentation in Robotic Surgeries](https://mlsanity.com/paper/2309.00957/) (2023)
- [Optimizing Latent Graph Representations of Surgical Scenes for Zero-Shot Domain Transfer](https://mlsanity.com/paper/2403.06953/) (2024)
- [Video-Instrument Synergistic Network for Referring Video Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2308.09475/) (2023)
- [Motion-Corrected Moving Average: Including Post-Hoc Temporal Information for Improved Video Segmentation](https://mlsanity.com/paper/2403.03120/) (2024)
- [Label-supervised surgical instrument segmentation using temporal equivariance and semantic continuity](https://mlsanity.com/paper/2403.09551/) (2024)
- [Amodal Segmentation for Laparoscopic Surgery Video Instruments](https://mlsanity.com/paper/2408.01067/) (2024)
- [Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions](https://mlsanity.com/paper/2606.26634/) (2026)

## Abstract (arXiv)

Automatic surgical instrument segmentation of endoscopic images is a crucial building block of many computer-assistance applications for minimally invasive surgery. So far, state-of-the-art approaches completely rely on the availability of a ground-truth supervision signal, obtained via manual annotation, thus expensive to collect at large scale. In this paper, we present FUN-SIS, a Fully-UNsupervised approach for binary Surgical Instrument Segmentation. FUN-SIS trains a per-frame segmentation model on completely unlabelled endoscopic videos, by solely relying on implicit motion information and instrument shape-priors. We define shape-priors as realistic segmentation masks of the instruments, not necessarily coming from the same dataset/domain as the videos. The shape-priors can be collected in various and convenient ways, such as recycling existing annotations from other datasets. We leverage them as part of a novel generative-adversarial approach, allowing to perform unsupervised instrument segmentation of optical-flow images during training. We then use the obtained instrument masks as pseudo-labels in order to train a per-frame segmentation model; to this aim, we develop a learning-from-noisy-labels architecture, designed to extract a clean supervision signal from these pseudo-labels, leveraging their peculiar noise properties. We validate the proposed contributions on three surgical datasets, including the MICCAI 2017 EndoVis Robotic Instrument Segmentation Challenge dataset. The obtained fully-unsupervised results for surgical instrument segmentation are almost on par with the ones of fully-supervised state-of-the-art approaches. This suggests the tremendous potential of the proposed method to leverage the great amount of unlabelled data produced in the context of minimally invasive surgery.