# From Forks to Forceps: A New Framework for Instance Segmentation of Surgical Instruments

arXiv: https://arxiv.org/abs/2211.16200 · HTML: https://mlsanity.com/paper/2211.16200/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 54 references that are in the map.
- First posted on arXiv 26 Nov 2022.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [Mask Scoring R-CNN](https://mlsanity.com/paper/1903.00241/) (2019)
- [DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution](https://mlsanity.com/paper/2006.02334/) (2020)
- [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019)
- [Instances as Queries](https://mlsanity.com/paper/2105.01928/) (2021)
- [The Devil is in Classification: A Simple Framework for Long-tail Object Detection and Instance Segmentation](https://mlsanity.com/paper/2007.11978/) (2020)
- [Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning](https://mlsanity.com/paper/1803.01207/) (2018)
- [FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentation](https://mlsanity.com/paper/2102.12867/) (2021)
- [Boundary-preserving Mask R-CNN](https://mlsanity.com/paper/2007.08921/) (2020)
- [ISINet: An Instance-Based Approach for Surgical Instrument Segmentation](https://mlsanity.com/paper/2007.05533/) (2020)
- [Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video](https://mlsanity.com/paper/1907.07899/) (2019)
- [Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks](https://mlsanity.com/paper/1802.08774/) (2018)
- [Incremental Few-Shot Instance Segmentation](https://mlsanity.com/paper/2105.05312/) (2021)
- [Convolutional Feature Masking for Joint Object and Stuff Segmentation](https://mlsanity.com/paper/1412.1283/) (2014)
- [TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2202.08453/) (2022)
- [SCNet: Training Inference Sample Consistency for Instance Segmentation](https://mlsanity.com/paper/2012.10150/) (2020)
- [Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery](https://mlsanity.com/paper/1805.02475/) (2018)
- [Learning Motion Flows for Semi-supervised Instrument Segmentation from Robotic Surgical Video](https://mlsanity.com/paper/2007.02501/) (2020)
- [RAUNet: Residual Attention U-Net for Semantic Segmentation of Cataract Surgical Instruments](https://mlsanity.com/paper/1909.10360/) (2019)
- [U-NetPlus: A Modified Encoder-Decoder U-Net Architecture for Semantic and Instance Segmentation of Surgical Instrument](https://mlsanity.com/paper/1902.08994/) (2019)
- [Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning](https://mlsanity.com/paper/2007.11319/) (2020)
- [Robust Medical Instrument Segmentation Challenge 2019](https://mlsanity.com/paper/2003.10299/) (2020)
- [CaDIS: Cataract Dataset for Image Segmentation](https://mlsanity.com/paper/1906.11586/) (2019)

## Built on by (most cited first, 30 of 32)

- [SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation](https://mlsanity.com/paper/2308.08746/) (2023)
- [Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruning](https://mlsanity.com/paper/2408.07931/) (2024)
- [AdaptiveSAM: Towards Efficient Tuning of SAM for Surgical Scene Segmentation](https://mlsanity.com/paper/2308.03726/) (2023)
- [SAM Meets Robotic Surgery: An Empirical Study on Generalization, Robustness and Adaptation](https://mlsanity.com/paper/2308.07156/) (2023)
- [Text Promptable Surgical Instrument Segmentation with Vision-Language Models](https://mlsanity.com/paper/2306.09244/) (2023)
- [SAM Meets Robotic Surgery: An Empirical Study in Robustness Perspective](https://mlsanity.com/paper/2304.14674/) (2023)
- [SAM 2 in Robotic Surgery: An Empirical Evaluation for Robustness and Generalization in Surgical Video Segmentation](https://mlsanity.com/paper/2408.04593/) (2024)
- [Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art](https://mlsanity.com/paper/2304.13014/) (2023)
- [CholecInstanceSeg: A Tool Instance Segmentation Dataset for Laparoscopic Surgery](https://mlsanity.com/paper/2406.16039/) (2024)
- [ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding](https://mlsanity.com/paper/2407.19435/) (2024)
- [SurgicalPart-SAM: Part-to-Whole Collaborative Prompting for Surgical Instrument Segmentation](https://mlsanity.com/paper/2312.14481/) (2023)
- [Surgical-DeSAM: Decoupling SAM for Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2404.14040/) (2024)
- [Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review](https://mlsanity.com/paper/2502.14886/) (2025)
- [Does DINOv3 Set a New Medical Vision Standard? Benchmarking 2D and 3D Classification, Segmentation, and Registration](https://mlsanity.com/paper/2509.06467/) (2025)
- [Multitask Learning in Minimally Invasive Surgical Vision: A Review](https://mlsanity.com/paper/2401.08256/) (2024)
- [Systematic Evaluation and Guidelines for Segment Anything Model in Surgical Video Analysis](https://mlsanity.com/paper/2501.00525/) (2024)
- [EgoSurgery-HTS: A Dataset for Egocentric Hand-Tool Segmentation in Open Surgery Videos](https://mlsanity.com/paper/2503.18755/) (2025)
- [Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data](https://mlsanity.com/paper/2501.15326/) (2025)
- [Segment Anything for Video: A Comprehensive Review of Video Object Segmentation and Tracking from Past to Future](https://mlsanity.com/paper/2507.22792/) (2025)
- [TSMS-SAM2: Multi-scale Temporal Sampling Augmentation and Memory-Splitting Pruning for Promptable Video Object Segmentation and Tracking in Surgical Scenarios](https://mlsanity.com/paper/2508.05829/) (2025)
- [More than Segmentation: Benchmarking SAM 3 for Segmentation, 3D Perception, and Reconstruction in Robotic Surgery](https://mlsanity.com/paper/2512.07596/) (2025)
- [Hierarchical Prototype-Memory Adaptation of SAM for Surgical Instrument Segmentation](https://mlsanity.com/paper/2608.24541/) (2026)
- [Label-supervised surgical instrument segmentation using temporal equivariance and semantic continuity](https://mlsanity.com/paper/2403.09551/) (2024)
- [Rethinking Text-Promptable Surgical Instrument Segmentation with Robust Framework](https://mlsanity.com/paper/2411.12199/) (2024)
- [MT3DNet: Multi-Task learning Network for 3D Surgical Scene Reconstruction](https://mlsanity.com/paper/2412.03928/) (2024)
- [SurgPIS: Surgical-instrument-level Instances and Part-level Semantics for Weakly-supervised Part-aware Instance Segmentation](https://mlsanity.com/paper/2507.19592/) (2025)
- [NeeCo: Image Synthesis of Novel Instrument States Based on Dynamic and Deformable 3D Gaussian Reconstruction](https://mlsanity.com/paper/2508.07897/) (2025)
- [FASL-Seg: Anatomy and Tool Segmentation of Surgical Scenes](https://mlsanity.com/paper/2509.06159/) (2025)
- [ReMeDI: Refined Memory for Disambiguation of Identities with SAM3 in Surgical Segmentation](https://mlsanity.com/paper/2512.16880/) (2025)
- [Toward Real-Time Surgical Scene Segmentation via a Spike-Driven Video Transformer with Spike-Informed Pretraining](https://mlsanity.com/paper/2512.21284/) (2025)

## Abstract (arXiv)

Minimally invasive surgeries and related applications demand surgical tool classification and segmentation at the instance level. Surgical tools are similar in appearance and are long, thin, and handled at an angle. The fine-tuning of state-of-the-art (SOTA) instance segmentation models trained on natural images for instrument segmentation has difficulty discriminating instrument classes. Our research demonstrates that while the bounding box and segmentation mask are often accurate, the classification head mis-classifies the class label of the surgical instrument. We present a new neural network framework that adds a classification module as a new stage to existing instance segmentation models. This module specializes in improving the classification of instrument masks generated by the existing model. The module comprises multi-scale mask attention, which attends to the instrument region and masks the distracting background features. We propose training our classifier module using metric learning with arc loss to handle low inter-class variance of surgical instruments. We conduct exhaustive experiments on the benchmark datasets EndoVis2017 and EndoVis2018. We demonstrate that our method outperforms all (more than 18) SOTA methods compared with, and improves the SOTA performance by at least 12 points (20%) on the EndoVis2017 benchmark challenge and generalizes effectively across the datasets.