# Real-Time Segmentation of Non-Rigid Surgical Tools based on Deep Learning and Tracking

arXiv: https://arxiv.org/abs/2009.03016 · HTML: https://mlsanity.com/paper/2009.03016/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2020.

## Built on by (most cited first, 15 of 15)

- [Image Compositing for Segmentation of Surgical Tools without Manual Annotations](https://mlsanity.com/paper/2102.09528/) (2021)
- [Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art](https://mlsanity.com/paper/2304.13014/) (2023)
- [Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review](https://mlsanity.com/paper/2502.14886/) (2025)
- [How can we learn (more) from challenges? A statistical approach to driving future algorithm development](https://mlsanity.com/paper/2106.09302/) (2021)
- [A comprehensive survey on recent deep learning-based methods applied to surgical data](https://mlsanity.com/paper/2209.01435/) (2022)
- [Revisiting Surgical Instrument Segmentation Without Human Intervention: A Graph Partitioning View](https://mlsanity.com/paper/2408.14789/) (2024)
- [AMNCutter: Affinity-Attention-Guided Multi-View Normalized Cutter for Unsupervised Surgical Instrument Segmentation](https://mlsanity.com/paper/2411.03695/) (2024)
- [Robotic Scene Segmentation with Memory Network for Runtime Surgical Context Inference](https://mlsanity.com/paper/2308.12789/) (2023)
- [Identifying Surgical Instruments in Laparoscopy Using Deep Learning Instance Segmentation](https://mlsanity.com/paper/2508.21399/) (2025)
- [Deep Feature Tracker: A Novel Application for Deep Convolutional Neural Networks](https://mlsanity.com/paper/2108.00105/) (2021)
- [LapSeg3D: Weakly Supervised Semantic Segmentation of Point Clouds Representing Laparoscopic Scenes](https://mlsanity.com/paper/2207.07418/) (2022)
- [Robust Surgical Tools Detection in Endoscopic Videos with Noisy Data](https://mlsanity.com/paper/2307.01232/) (2023)
- [Plug-in for visualizing 3D tool tracking from videos of Minimally Invasive Surgeries](https://mlsanity.com/paper/2401.09472/) (2024)
- [Motion-Corrected Moving Average: Including Post-Hoc Temporal Information for Improved Video Segmentation](https://mlsanity.com/paper/2403.03120/) (2024)
- [orGAN: A Synthetic Data Augmentation Pipeline for Simultaneous Generation of Surgical Images and Ground Truth Labels](https://mlsanity.com/paper/2506.14303/) (2025)

## Abstract (arXiv)

Real-time tool segmentation is an essential component in computer-assisted surgical systems. We propose a novel real-time automatic method based on Fully Convolutional Networks (FCN) and optical flow tracking. Our method exploits the ability of deep neural networks to produce accurate segmentations of highly deformable parts along with the high speed of optical flow. Furthermore, the pre-trained FCN can be fine-tuned on a small amount of medical images without the need to hand-craft features. We validated our method using existing and new benchmark datasets, covering both ex vivo and in vivo real clinical cases where different surgical instruments are employed. Two versions of the method are presented, non-real-time and real-time. The former, using only deep learning, achieves a balanced accuracy of 89.6% on a real clinical dataset, outperforming the (non-real-time) state of the art by 3.8% points. The latter, a combination of deep learning with optical flow tracking, yields an average balanced accuracy of 78.2% across all the validated datasets.