# ToolTipNet: A Segmentation-Driven Deep Learning Baseline for Surgical Instrument Tip Detection

arXiv: https://arxiv.org/abs/2504.09700 · HTML: https://mlsanity.com/paper/2504.09700/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 13 Apr 2025.

## Builds on (most cited first, 15 of 15)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning](https://mlsanity.com/paper/2501.06994/) (2025)
- [Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation](https://mlsanity.com/paper/2502.20391/) (2025)
- [Single-view robot pose and joint angle estimation via render & compare](https://mlsanity.com/paper/2104.09359/) (2021)
- [Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer](https://mlsanity.com/paper/2302.14332/) (2023)
- [SurgPose: a Dataset for Articulated Robotic Surgical Tool Pose Estimation and Tracking](https://mlsanity.com/paper/2502.11534/) (2025)
- [Augmenting Efficient Real-time Surgical Instrument Segmentation in Video with Point Tracking and Segment Anything](https://mlsanity.com/paper/2403.08003/) (2024)
- [Differentiable Rendering-based Pose Estimation for Surgical Robotic Instruments](https://mlsanity.com/paper/2503.05953/) (2025)
- [Video-based Surgical Tool-tip and Keypoint Tracking using Multi-frame Context-driven Deep Learning Models](https://mlsanity.com/paper/2501.18361/) (2025)
- [Instrument-Splatting: Controllable Photorealistic Reconstruction of Surgical Instruments Using Gaussian Splatting](https://mlsanity.com/paper/2503.04082/) (2025)
- [Deep Attention Based Semi-Supervised 2D-Pose Estimation for Surgical Instruments](https://mlsanity.com/paper/1912.04618/) (2019)
- [Automatic Search for Photoacoustic Marker Using Automated Transrectal Ultrasound](https://mlsanity.com/paper/2307.10546/) (2023)
- [GISR: Geometric Initialization and Silhouette-based Refinement for Single-View Robot Pose and Configuration Estimation](https://mlsanity.com/paper/2405.04890/) (2024)

## Built on by (most cited first, 1 of 1)

- [SurgCalib: Gaussian Splatting-Based Hand-Eye Calibration for Robot-Assisted Minimally Invasive Surgery](https://mlsanity.com/paper/2603.08983/) (2026)

## Abstract (arXiv)

In robot-assisted laparoscopic radical prostatectomy (RALP), the location of the instrument tip is important to register the ultrasound frame with the laparoscopic camera frame. A long-standing limitation is that the instrument tip position obtained from the da Vinci API is inaccurate and requires hand-eye calibration. Thus, directly computing the position of the tool tip in the camera frame using the vision-based method becomes an attractive solution. Besides, surgical instrument tip detection is the key component of other tasks, like surgical skill assessment and surgery automation. However, this task is challenging due to the small size of the tool tip and the articulation of the surgical instrument. Surgical instrument segmentation becomes relatively easy due to the emergence of the Segmentation Foundation Model, i.e., Segment Anything. Based on this advancement, we explore the deep learning-based surgical instrument tip detection approach that takes the part-level instrument segmentation mask as input. Comparison experiments with a hand-crafted image-processing approach demonstrate the superiority of the proposed method on simulated and real datasets.