# Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery

arXiv: https://arxiv.org/abs/1805.02475 · HTML: https://mlsanity.com/paper/1805.02475/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2018.

## Built on by (most cited first, 30 of 37)

- [Pixel-Wise Recognition for Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2401.11174/) (2024)
- [MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation](https://mlsanity.com/paper/2303.09514/) (2023)
- [From Forks to Forceps: A New Framework for Instance Segmentation of Surgical Instruments](https://mlsanity.com/paper/2211.16200/) (2022)
- [Kvasir-VQA: A Text-Image Pair GI Tract Dataset](https://mlsanity.com/paper/2409.01437/) (2024)
- [FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation](https://mlsanity.com/paper/2202.08141/) (2022)
- [Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art](https://mlsanity.com/paper/2304.13014/) (2023)
- [Real-time Instance Segmentation of Surgical Instruments using Attention and Multi-scale Feature Fusion](https://mlsanity.com/paper/2111.04911/) (2021)
- [NanoNet: Real-Time Polyp Segmentation in Video Capsule Endoscopy and Colonoscopy](https://mlsanity.com/paper/2104.11138/) (2021)
- [CholecInstanceSeg: A Tool Instance Segmentation Dataset for Laparoscopic Surgery](https://mlsanity.com/paper/2406.16039/) (2024)
- [SurgPose: a Dataset for Articulated Robotic Surgical Tool Pose Estimation and Tracking](https://mlsanity.com/paper/2502.11534/) (2025)
- [OpenMIBOOD: Open Medical Imaging Benchmarks for Out-Of-Distribution Detection](https://mlsanity.com/paper/2503.16247/) (2025)
- [VidLPRO: A underlineVideo-underlineLanguage underlinePre-training Framework for underlineRobotic and Laparoscopic Surgery](https://mlsanity.com/paper/2409.04732/) (2024)
- [Online 3D reconstruction and dense tracking in endoscopic videos](https://mlsanity.com/paper/2409.06037/) (2024)
- [Simulation-to-Real domain adaptation with teacher-student learning for endoscopic instrument segmentation](https://mlsanity.com/paper/2103.01593/) (2021)
- [Video Dataset for Surgical Phase, Keypoint, and Instrument Recognition in Laparoscopic Surgery (PhaKIR)](https://mlsanity.com/paper/2511.06549/) (2025)
- [Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challenge](https://mlsanity.com/paper/2507.16559/) (2025)
- [Full or Weak annotations? An adaptive strategy for budget-constrained annotation campaigns](https://mlsanity.com/paper/2303.11678/) (2023)
- [Augmenting Efficient Real-time Surgical Instrument Segmentation in Video with Point Tracking and Segment Anything](https://mlsanity.com/paper/2403.08003/) (2024)
- [Assessing YOLACT++ for real time and robust instance segmentation of medical instruments in endoscopic procedures](https://mlsanity.com/paper/2103.15997/) (2021)
- [How can we learn (more) from challenges? A statistical approach to driving future algorithm development](https://mlsanity.com/paper/2106.09302/) (2021)
- [Correlation-aware active learning for surgery video segmentation](https://mlsanity.com/paper/2311.08811/) (2023)
- [Exploring Deep Learning Methods for Real-Time Surgical Instrument Segmentation in Laparoscopy](https://mlsanity.com/paper/2107.02319/) (2021)
- [A comprehensive survey on recent deep learning-based methods applied to surgical data](https://mlsanity.com/paper/2209.01435/) (2022)
- [Phase-Informed Tool Segmentation for Manual Small-Incision Cataract Surgery](https://mlsanity.com/paper/2411.16794/) (2024)
- [EgoSurgery-HTS: A Dataset for Egocentric Hand-Tool Segmentation in Open Surgery Videos](https://mlsanity.com/paper/2503.18755/) (2025)
- [Robotic Scene Segmentation with Memory Network for Runtime Surgical Context Inference](https://mlsanity.com/paper/2308.12789/) (2023)
- [Artificial Intelligence for Biomedical Video Generation](https://mlsanity.com/paper/2411.07619/) (2024)
- [Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development](https://mlsanity.com/paper/2603.27460/) (2026)
- [Deep Homography Estimation in Dynamic Surgical Scenes for Laparoscopic Camera Motion Extraction](https://mlsanity.com/paper/2109.15098/) (2021)
- [mvHOTA: A multi-view higher order tracking accuracy metric to measure spatial and temporal associations in multi-point detection](https://mlsanity.com/paper/2206.09372/) (2022)

## Abstract (arXiv)

Intraoperative segmentation and tracking of minimally invasive instruments is a prerequisite for computer- and robotic-assisted surgery. Since additional hardware like tracking systems or the robot encoders are cumbersome and lack accuracy, surgical vision is evolving as promising techniques to segment and track the instruments using only the endoscopic images. However, what is missing so far are common image data sets for consistent evaluation and benchmarking of algorithms against each other. The paper presents a comparative validation study of different vision-based methods for instrument segmentation and tracking in the context of robotic as well as conventional laparoscopic surgery. The contribution of the paper is twofold: we introduce a comprehensive validation data set that was provided to the study participants and present the results of the comparative validation study. Based on the results of the validation study, we arrive at the conclusion that modern deep learning approaches outperform other methods in instrument segmentation tasks, but the results are still not perfect. Furthermore, we show that merging results from different methods actually significantly increases accuracy in comparison to the best stand-alone method. On the other hand, the results of the instrument tracking task show that this is still an open challenge, especially during challenging scenarios in conventional laparoscopic surgery.