# Endo-Sim2Real: Consistency learning-based domain adaptation for instrument segmentation

arXiv: https://arxiv.org/abs/2007.11514 · HTML: https://mlsanity.com/paper/2007.11514/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2020.

## Built on by (most cited first, 20 of 20)

- [Deep learning for unsupervised domain adaptation in medical imaging: Recent advancements and future perspectives](https://mlsanity.com/paper/2308.01265/) (2023)
- [Long-Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Data](https://mlsanity.com/paper/2103.17204/) (2021)
- [FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation](https://mlsanity.com/paper/2202.08141/) (2022)
- [Class-Incremental Domain Adaptation with Smoothing and Calibration for Surgical Report Generation](https://mlsanity.com/paper/2107.11091/) (2021)
- [Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art](https://mlsanity.com/paper/2304.13014/) (2023)
- [SimT: Handling Open-set Noise for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2203.15202/) (2022)
- [Surgical Visual Domain Adaptation: Results from the MICCAI 2020 SurgVisDom Challenge](https://mlsanity.com/paper/2102.13644/) (2021)
- [One to Many: Adaptive Instrument Segmentation via Meta Learning and Dynamic Online Adaptation in Robotic Surgical Video](https://mlsanity.com/paper/2103.12988/) (2021)
- [Towards Artificial General Intelligence (AGI) in the Internet of Things (IoT): Opportunities and Challenges](https://mlsanity.com/paper/2309.07438/) (2023)
- [Simulation-to-Real domain adaptation with teacher-student learning for endoscopic instrument segmentation](https://mlsanity.com/paper/2103.01593/) (2021)
- [Synthetic Data in Healthcare](https://mlsanity.com/paper/2304.03243/) (2023)
- [SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Models](https://mlsanity.com/paper/2412.02332/) (2024)
- [Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review](https://mlsanity.com/paper/2502.14886/) (2025)
- [Data efficient deep learning for medical image analysis: A survey](https://mlsanity.com/paper/2310.06557/) (2023)
- [A comprehensive survey on recent deep learning-based methods applied to surgical data](https://mlsanity.com/paper/2209.01435/) (2022)
- [Domain Adaptive Robotic Gesture Recognition with Unsupervised Kinematic-Visual Data Alignment](https://mlsanity.com/paper/2103.04075/) (2021)
- [SAF-IS: a Spatial Annotation Free Framework for Instance Segmentation of Surgical Tools](https://mlsanity.com/paper/2309.01723/) (2023)
- [Visual-Kinematics Graph Learning for Procedure-agnostic Instrument Tip Segmentation in Robotic Surgeries](https://mlsanity.com/paper/2309.00957/) (2023)
- [Sim2Real in endoscopy segmentation with a novel structure aware image translation](https://mlsanity.com/paper/2505.02654/) (2025)
- [NeeCo: Image Synthesis of Novel Instrument States Based on Dynamic and Deformable 3D Gaussian Reconstruction](https://mlsanity.com/paper/2508.07897/) (2025)

## Abstract (arXiv)

Surgical tool segmentation in endoscopic videos is an important component of computer assisted interventions systems. Recent success of image-based solutions using fully-supervised deep learning approaches can be attributed to the collection of big labeled datasets. However, the annotation of a big dataset of real videos can be prohibitively expensive and time consuming. Computer simulations could alleviate the manual labeling problem, however, models trained on simulated data do not generalize to real data. This work proposes a consistency-based framework for joint learning of simulated and real (unlabeled) endoscopic data to bridge this performance generalization issue. Empirical results on two data sets (15 videos of the Cholec80 and EndoVis'15 dataset) highlight the effectiveness of the proposed Endo-Sim2Real method for instrument segmentation. We compare the segmentation of the proposed approach with state-of-the-art solutions and show that our method improves segmentation both in terms of quality and quantity.