# Reducing Annotating Load: Active Learning with Synthetic Images in Surgical Instrument Segmentation

arXiv: https://arxiv.org/abs/2108.03534 · HTML: https://mlsanity.com/paper/2108.03534/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 60 references that are in the map.
- First posted on arXiv 7 Aug 2021.

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation](https://mlsanity.com/paper/1908.10454/) (2019)
- [Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection](https://mlsanity.com/paper/1708.01642/) (2017)
- [Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis](https://mlsanity.com/paper/1804.06353/) (2018)
- [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019)
- [A review of domain adaptation without target labels](https://mlsanity.com/paper/1901.05335/) (2019)
- [A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1910.02923/) (2019)
- [Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation](https://mlsanity.com/paper/1706.04737/) (2017)
- [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017)
- [Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning](https://mlsanity.com/paper/1803.01207/) (2018)
- [Bayesian Generative Active Deep Learning](https://mlsanity.com/paper/1904.11643/) (2019)
- [Learning to Segment via Cut-and-Paste](https://mlsanity.com/paper/1803.06414/) (2018)
- [Cost-Effective Active Learning for Melanoma Segmentation](https://mlsanity.com/paper/1711.09168/) (2017)
- [Exploiting the potential of unlabeled endoscopic video data with self-supervised learning](https://mlsanity.com/paper/1711.09726/) (2017)
- [Image Compositing for Segmentation of Surgical Tools without Manual Annotations](https://mlsanity.com/paper/2102.09528/) (2021)
- [Learning Where to Look While Tracking Instruments in Robot-assisted Surgery](https://mlsanity.com/paper/1907.00214/) (2019)
- [Temporal coherence-based self-supervised learning for laparoscopic workflow analysis](https://mlsanity.com/paper/1806.06811/) (2018)
- [Deep Active Learning for Joint Classification & Segmentation with Weak Annotator](https://mlsanity.com/paper/2010.04889/) (2020)
- [Unsupervised Surgical Instrument Segmentation via Anchor Generation and Semantic Diffusion](https://mlsanity.com/paper/2008.11946/) (2020)
- [Reducing bias and increasing utility by federated generative modeling of medical images using a centralized adversary](https://mlsanity.com/paper/2101.07235/) (2021)
- [A Tour of Unsupervised Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1812.07715/) (2018)
- [LC-GAN: Image-to-image Translation Based on Generative Adversarial Network for Endoscopic Images](https://mlsanity.com/paper/2003.04949/) (2020)
- [A Kinematic Bottleneck Approach For Pose Regression of Flexible Surgical Instruments directly from Images](https://mlsanity.com/paper/2103.00586/) (2021)
- [Towards Better Surgical Instrument Segmentation in Endoscopic Vision: Multi-Angle Feature Aggregation and Contour Supervision](https://mlsanity.com/paper/2002.10675/) (2020)

## Built on by (most cited first, 6 of 6)

- [Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review](https://mlsanity.com/paper/2502.14886/) (2025)
- [Correlation-aware active learning for surgery video segmentation](https://mlsanity.com/paper/2311.08811/) (2023)
- [Spatio-Temporal Representation Decoupling and Enhancement for Federated Instrument Segmentation in Surgical Videos](https://mlsanity.com/paper/2506.23759/) (2025)
- [A comprehensive survey on recent deep learning-based methods applied to surgical data](https://mlsanity.com/paper/2209.01435/) (2022)
- [Active Learning with Context Sampling and One-vs-Rest Entropy for Semantic Segmentation](https://mlsanity.com/paper/2412.06470/) (2024)
- [SurgFed: Language-guided Multi-Task Federated Learning for Surgical Video Understanding](https://mlsanity.com/paper/2603.09496/) (2026)

## Abstract (arXiv)

Accurate instrument segmentation in endoscopic vision of robot-assisted surgery is challenging due to reflection on the instruments and frequent contacts with tissue. Deep neural networks (DNN) show competitive performance and are in favor in recent years. However, the hunger of DNN for labeled data poses a huge workload of annotation. Motivated by alleviating this workload, we propose a general embeddable method to decrease the usage of labeled real images, using active generated synthetic images. In each active learning iteration, the most informative unlabeled images are first queried by active learning and then labeled. Next, synthetic images are generated based on these selected images. The instruments and backgrounds are cropped out and randomly combined with each other with blending and fusion near the boundary. The effectiveness of the proposed method is validated on 2 sinus surgery datasets and 1 intraabdominal surgery dataset. The results indicate a considerable improvement in performance, especially when the budget for annotation is small. The effectiveness of different types of synthetic images, blending methods, and external background are also studied. All the code is open-sourced at: https://github.com/HaonanPeng/active_syn_generator.