# Concurrent Segmentation and Localization for Tracking of Surgical Instruments

arXiv: https://arxiv.org/abs/1703.10701 · HTML: https://mlsanity.com/paper/1703.10701/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2017.

## Built on by (most cited first, 24 of 24)

- [4D-OR: Semantic Scene Graphs for OR Domain Modeling](https://mlsanity.com/paper/2203.11937/) (2022)
- [MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation](https://mlsanity.com/paper/2303.09514/) (2023)
- [OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretraining](https://mlsanity.com/paper/2411.15421/) (2024)
- [ST-MTL: Spatio-Temporal Multitask Learning Model to Predict Scanpath While Tracking Instruments in Robotic Surgery](https://mlsanity.com/paper/2112.08189/) (2021)
- [FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation](https://mlsanity.com/paper/2202.08141/) (2022)
- [Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art](https://mlsanity.com/paper/2304.13014/) (2023)
- [A Kinematic Bottleneck Approach For Pose Regression of Flexible Surgical Instruments directly from Images](https://mlsanity.com/paper/2103.00586/) (2021)
- [Simulation-to-Real domain adaptation with teacher-student learning for endoscopic instrument segmentation](https://mlsanity.com/paper/2103.01593/) (2021)
- [Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review](https://mlsanity.com/paper/2502.14886/) (2025)
- [SegMatch: A semi-supervised learning method for surgical instrument segmentation](https://mlsanity.com/paper/2308.05232/) (2023)
- [Multitask Learning in Minimally Invasive Surgical Vision: A Review](https://mlsanity.com/paper/2401.08256/) (2024)
- [Real-time landmark detection for precise endoscopic submucosal dissection via shape-aware relation network](https://mlsanity.com/paper/2111.04733/) (2021)
- [A comprehensive survey on recent deep learning-based methods applied to surgical data](https://mlsanity.com/paper/2209.01435/) (2022)
- [Revisiting Surgical Instrument Segmentation Without Human Intervention: A Graph Partitioning View](https://mlsanity.com/paper/2408.14789/) (2024)
- [AMNCutter: Affinity-Attention-Guided Multi-View Normalized Cutter for Unsupervised Surgical Instrument Segmentation](https://mlsanity.com/paper/2411.03695/) (2024)
- [Temporally Guided Articulated Hand Pose Tracking in Surgical Videos](https://mlsanity.com/paper/2101.04281/) (2021)
- [SAF-IS: a Spatial Annotation Free Framework for Instance Segmentation of Surgical Tools](https://mlsanity.com/paper/2309.01723/) (2023)
- [Strategies to Improve Real-World Applicability of Laparoscopic Anatomy Segmentation Models](https://mlsanity.com/paper/2403.17192/) (2024)
- [Identifying Surgical Instruments in Laparoscopy Using Deep Learning Instance Segmentation](https://mlsanity.com/paper/2508.21399/) (2025)
- [Segmenting Medical Instruments in Minimally Invasive Surgeries using AttentionMask](https://mlsanity.com/paper/2203.11358/) (2022)
- [Real-Time Surgical Instrument Defect Detection via Non-Destructive Testing](https://mlsanity.com/paper/2510.14525/) (2025)
- [Towards Comprehensive Real-Time Scene Understanding in Ophthalmic Surgery through Multimodal Image Fusion](https://mlsanity.com/paper/2603.25555/) (2026)
- [SurfSurg6D: Geometry Consistent Dense Correspondence for Textureless Surgical Instrument Pose Estimation](https://mlsanity.com/paper/2605.25598/) (2026)
- [Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions](https://mlsanity.com/paper/2606.26634/) (2026)

## Abstract (arXiv)

Real-time instrument tracking is a crucial requirement for various computer-assisted interventions. In order to overcome problems such as specular reflections and motion blur, we propose a novel method that takes advantage of the interdependency between localization and segmentation of the surgical tool. In particular, we reformulate the 2D instrument pose estimation as heatmap regression and thereby enable a concurrent, robust and near real-time regression of both tasks via deep learning. As demonstrated by our experimental results, this modeling leads to a significantly improved performance than directly regressing the tool position and allows our method to outperform the state of the art on a Retinal Microsurgery benchmark and the MICCAI EndoVis Challenge 2015.