# Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning

arXiv: https://arxiv.org/abs/2007.11319 · HTML: https://mlsanity.com/paper/2007.11319/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2020.

## Built on by (most cited first, 23 of 23)

- [From Forks to Forceps: A New Framework for Instance Segmentation of Surgical Instruments](https://mlsanity.com/paper/2211.16200/) (2022)
- [SAM Meets Robotic Surgery: An Empirical Study on Generalization, Robustness and Adaptation](https://mlsanity.com/paper/2308.07156/) (2023)
- [SAM Meets Robotic Surgery: An Empirical Study in Robustness Perspective](https://mlsanity.com/paper/2304.14674/) (2023)
- [SAM 2 in Robotic Surgery: An Empirical Evaluation for Robustness and Generalization in Surgical Video Segmentation](https://mlsanity.com/paper/2408.04593/) (2024)
- [ST-MTL: Spatio-Temporal Multitask Learning Model to Predict Scanpath While Tracking Instruments in Robotic Surgery](https://mlsanity.com/paper/2112.08189/) (2021)
- [Brain Tumor Segmentation and Survival Prediction using 3D Attention UNet](https://mlsanity.com/paper/2104.00985/) (2021)
- [Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art](https://mlsanity.com/paper/2304.13014/) (2023)
- [XVO: Generalized Visual Odometry via Cross-Modal Self-Training](https://mlsanity.com/paper/2309.16772/) (2023)
- [CaRTS: Causality-driven Robot Tool Segmentation from Vision and Kinematics Data](https://mlsanity.com/paper/2203.09475/) (2022)
- [CholecInstanceSeg: A Tool Instance Segmentation Dataset for Laparoscopic Surgery](https://mlsanity.com/paper/2406.16039/) (2024)
- [One to Many: Adaptive Instrument Segmentation via Meta Learning and Dynamic Online Adaptation in Robotic Surgical Video](https://mlsanity.com/paper/2103.12988/) (2021)
- [Deep Learning for Surgical Instrument Recognition and Segmentation in Robotic-Assisted Surgeries: A Systematic Review](https://mlsanity.com/paper/2410.07269/) (2024)
- [Future Frame Prediction for Robot-assisted Surgery](https://mlsanity.com/paper/2103.10308/) (2021)
- [Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review](https://mlsanity.com/paper/2502.14886/) (2025)
- [Glioblastoma Multiforme Prognosis: MRI Missing Modality Generation, Segmentation and Radiogenomic Survival Prediction](https://mlsanity.com/paper/2104.01149/) (2021)
- [Rethinking Causality-driven Robot Tool Segmentation with Temporal Constraints](https://mlsanity.com/paper/2212.00072/) (2022)
- [Multitask Learning in Minimally Invasive Surgical Vision: A Review](https://mlsanity.com/paper/2401.08256/) (2024)
- [A comprehensive survey on recent deep learning-based methods applied to surgical data](https://mlsanity.com/paper/2209.01435/) (2022)
- [Revisiting Surgical Instrument Segmentation Without Human Intervention: A Graph Partitioning View](https://mlsanity.com/paper/2408.14789/) (2024)
- [Rethinking RGB-D Fusion for Semantic Segmentation in Surgical Datasets](https://mlsanity.com/paper/2407.19714/) (2024)
- [More than Segmentation: Benchmarking SAM 3 for Segmentation, 3D Perception, and Reconstruction in Robotic Surgery](https://mlsanity.com/paper/2512.07596/) (2025)
- [Perception Reinforcement Using Auxiliary Learning Feature Fusion: A Modified Yolov8 for Head Detection](https://mlsanity.com/paper/2310.09492/) (2023)
- [One Patient's Annotation is Another One's Initialization: Towards Zero-Shot Surgical Video Segmentation with Cross-Patient Initialization](https://mlsanity.com/paper/2503.02228/) (2025)

## Abstract (arXiv)

Robot-assisted surgery is an emerging technology which has undergone rapid growth with the development of robotics and imaging systems. Innovations in vision, haptics and accurate movements of robot arms have enabled surgeons to perform precise minimally invasive surgeries. Real-time semantic segmentation of the robotic instruments and tissues is a crucial step in robot-assisted surgery. Accurate and efficient segmentation of the surgical scene not only aids in the identification and tracking of instruments but also provided contextual information about the different tissues and instruments being operated with. For this purpose, we have developed a light-weight cascaded convolutional neural network (CNN) to segment the surgical instruments from high-resolution videos obtained from a commercial robotic system. We propose a multi-resolution feature fusion module (MFF) to fuse the feature maps of different dimensions and channels from the auxiliary and main branch. We also introduce a novel way of combining auxiliary loss and adversarial loss to regularize the segmentation model. Auxiliary loss helps the model to learn low-resolution features, and adversarial loss improves the segmentation prediction by learning higher order structural information. The model also consists of a light-weight spatial pyramid pooling (SPP) unit to aggregate rich contextual information in the intermediate stage. We show that our model surpasses existing algorithms for pixel-wise segmentation of surgical instruments in both prediction accuracy and segmentation time of high-resolution videos.