# Rethinking Surgical Instrument Segmentation: A Background Image Can Be All You Need arXiv: https://arxiv.org/abs/2206.11804 ยท HTML: https://mlsanity.com/paper/2206.11804/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Builds on 13 of its 20 references that are in the map. - First posted on arXiv 23 Jun 2022. ## Builds on (most cited first, 13 of 13) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019) - [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018) - [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019) - [Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks](https://mlsanity.com/paper/1807.10225/) (2018) - [2018 Robotic Scene Segmentation Challenge](https://mlsanity.com/paper/2001.11190/) (2020) - [Synthesizing Diverse Lung Nodules Wherever Massively: 3D Multi-Conditional GAN-based CT Image Augmentation for Object Detection](https://mlsanity.com/paper/1906.04962/) (2019) - [Synthetic and Real Inputs for Tool Segmentation in Robotic Surgery](https://mlsanity.com/paper/2007.09107/) (2020) - [Image Compositing for Segmentation of Surgical Tools without Manual Annotations](https://mlsanity.com/paper/2102.09528/) (2021) - [When and how CNNs generalize to out-of-distribution category-viewpoint combinations](https://mlsanity.com/paper/2007.08032/) (2020) - [Class-Incremental Domain Adaptation with Smoothing and Calibration for Surgical Report Generation](https://mlsanity.com/paper/2107.11091/) (2021) - [High Tissue Contrast MRI Synthesis Using Multi-Stage Attention-GAN for Glioma Segmentation](https://mlsanity.com/paper/2006.05030/) (2020) - [Ensembles of GANs for synthetic training data generation](https://mlsanity.com/paper/2104.11797/) (2021) ## Built on by (most cited first, 16 of 16) - [SAM Meets Robotic Surgery: An Empirical Study on Generalization, Robustness and Adaptation](https://mlsanity.com/paper/2308.07156/) (2023) - [SAM Meets Robotic Surgery: An Empirical Study in Robustness Perspective](https://mlsanity.com/paper/2304.14674/) (2023) - [SAM 2 in Robotic Surgery: An Empirical Evaluation for Robustness and Generalization in Surgical Video Segmentation](https://mlsanity.com/paper/2408.04593/) (2024) - [Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art](https://mlsanity.com/paper/2304.13014/) (2023) - [S2ME: Spatial-Spectral Mutual Teaching and Ensemble Learning for Scribble-supervised Polyp Segmentation](https://mlsanity.com/paper/2306.00451/) (2023) - [ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding](https://mlsanity.com/paper/2407.19435/) (2024) - [Curriculum-Based Augmented Fourier Domain Adaptation for Robust Medical Image Segmentation](https://mlsanity.com/paper/2306.03511/) (2023) - [Does DINOv3 Set a New Medical Vision Standard? Benchmarking 2D and 3D Classification, Segmentation, and Registration](https://mlsanity.com/paper/2509.06467/) (2025) - [Semantic segmentation of surgical hyperspectral images under geometric domain shifts](https://mlsanity.com/paper/2303.10972/) (2023) - [Privacy-Preserving Synthetic Continual Semantic Segmentation for Robotic Surgery](https://mlsanity.com/paper/2402.05860/) (2024) - [Generalizing Surgical Instruments Segmentation to Unseen Domains with One-to-Many Synthesis](https://mlsanity.com/paper/2306.16285/) (2023) - [Strategies to Improve Real-World Applicability of Laparoscopic Anatomy Segmentation Models](https://mlsanity.com/paper/2403.17192/) (2024) - [Endo-TTAP: Robust Endoscopic Tissue Tracking via Multi-Facet Guided Attention and Hybrid Flow-point Supervision](https://mlsanity.com/paper/2503.22394/) (2025) - [More than Segmentation: Benchmarking SAM 3 for Segmentation, 3D Perception, and Reconstruction in Robotic Surgery](https://mlsanity.com/paper/2512.07596/) (2025) - [Video-Instrument Synergistic Network for Referring Video Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2308.09475/) (2023) - [CataractSAM-2: A Domain-Adapted Model for Anterior Segment Surgery Segmentation and Scalable Ground-Truth Annotation](https://mlsanity.com/paper/2603.21566/) (2026) ## Abstract (arXiv) Data diversity and volume are crucial to the success of training deep learning models, while in the medical imaging field, the difficulty and cost of data collection and annotation are especially huge. Specifically in robotic surgery, data scarcity and imbalance have heavily affected the model accuracy and limited the design and deployment of deep learning-based surgical applications such as surgical instrument segmentation. Considering this, we rethink the surgical instrument segmentation task and propose a one-to-many data generation solution that gets rid of the complicated and expensive process of data collection and annotation from robotic surgery. In our method, we only utilize a single surgical background tissue image and a few open-source instrument images as the seed images and apply multiple augmentations and blending techniques to synthesize amounts of image variations. In addition, we also introduce the chained augmentation mixing during training to further enhance the data diversities. The proposed approach is evaluated on the real datasets of the EndoVis-2018 and EndoVis-2017 surgical scene segmentation. Our empirical analysis suggests that without the high cost of data collection and annotation, we can achieve decent surgical instrument segmentation performance. Moreover, we also observe that our method can deal with novel instrument prediction in the deployment domain. We hope our inspiring results will encourage researchers to emphasize data-centric methods to overcome demanding deep learning limitations besides data shortage, such as class imbalance, domain adaptation, and incremental learning. Our code is available at https://github.com/lofrienger/Single_SurgicalScene_For_Segmentation.