# Robust Surgical Tool Tracking with Pixel-based Probabilities for Projected Geometric Primitives

arXiv: https://arxiv.org/abs/2403.04971 · HTML: https://mlsanity.com/paper/2403.04971/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 51 references that are in the map.
- First posted on arXiv 8 Mar 2024.

## Builds on (most cited first, 21 of 21)

- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Camera-to-Robot Pose Estimation from a Single Image](https://mlsanity.com/paper/1911.09231/) (2019)
- [SOLD2: Self-supervised Occlusion-aware Line Description and Detection](https://mlsanity.com/paper/2104.03362/) (2021)
- [SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics](https://mlsanity.com/paper/1909.05405/) (2019)
- [SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction](https://mlsanity.com/paper/2003.03472/) (2020)
- [Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer](https://mlsanity.com/paper/2302.14332/) (2023)
- [Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning](https://mlsanity.com/paper/2011.04813/) (2020)
- [Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Tracking](https://mlsanity.com/paper/2210.16674/) (2022)
- [Robot Autonomy for Surgery](https://mlsanity.com/paper/1707.03080/) (2017)
- [Robotic Tool Tracking under Partially Visible Kinematic Chain: A Unified Approach](https://mlsanity.com/paper/2102.06235/) (2021)
- [Probabilistic Articulated Real-Time Tracking for Robot Manipulation](https://mlsanity.com/paper/1610.04871/) (2016)
- [Autonomous Robotic Suction to Clear the Surgical Field for Hemostasis using Image-based Blood Flow Detection](https://mlsanity.com/paper/2010.08441/) (2020)
- [Real-to-Sim Registration of Deformable Soft Tissue with Position-Based Dynamics for Surgical Robot Autonomy](https://mlsanity.com/paper/2011.00800/) (2020)
- [Fast and Reliable Autonomous Surgical Debridement with Cable-Driven Robots Using a Two-Phase Calibration Procedure](https://mlsanity.com/paper/1709.06668/) (2017)
- [Learning to Localize, Grasp, and Hand Over Unmodified Surgical Needles](https://mlsanity.com/paper/2112.04071/) (2021)
- [Applying Depth-Sensing to Automated Surgical Manipulation with a da Vinci Robot](https://mlsanity.com/paper/2002.06302/) (2020)
- [Model-Predictive Control of Blood Suction for Surgical Hemostasis using Differentiable Fluid Simulations](https://mlsanity.com/paper/2102.01436/) (2021)
- [Markerless Suture Needle 6D Pose Tracking with Robust Uncertainty Estimation for Autonomous Minimally Invasive Robotic Surgery](https://mlsanity.com/paper/2109.12722/) (2021)
- [From Bench to Bedside: The First Live Robotic Surgery on the dVRK to Enable Remote Telesurgery with Motion Scaling](https://mlsanity.com/paper/2109.12177/) (2021)
- [Real-time Data Driven Precision Estimator for RAVEN-II Surgical Robot End Effector Position](https://mlsanity.com/paper/1910.06425/) (2019)
- [Real-Time Constrained 6D Object-Pose Tracking of An In-Hand Suture Needle for Minimally Invasive Robotic Surgery](https://mlsanity.com/paper/2210.11973/) (2022)

## Built on by (most cited first, 4 of 4)

- [Efficient Surgical Robotic Instrument Pose Reconstruction in Real World Conditions Using Unified Feature Detection](https://mlsanity.com/paper/2510.03532/) (2025)
- [Real-time Rendering-based Surgical Instrument Tracking via Evolutionary Optimization](https://mlsanity.com/paper/2603.11404/) (2026)
- [arg-VU: Affordance Reasoning with Physics-Aware 3D Geometry for Visual Understanding in Robotic Surgery](https://mlsanity.com/paper/2603.26814/) (2026)
- [Robust Surgical Robotic Instrument Tracking via Sequential Multi-Cue Fusion and Sim-to-Real Self-Training](https://mlsanity.com/paper/2610.05491/) (2026)

## Abstract (arXiv)

Controlling robotic manipulators via visual feedback requires a known coordinate frame transformation between the robot and the camera. Uncertainties in mechanical systems as well as camera calibration create errors in this coordinate frame transformation. These errors result in poor localization of robotic manipulators and create a significant challenge for applications that rely on precise interactions between manipulators and the environment. In this work, we estimate the camera-to-base transform and joint angle measurement errors for surgical robotic tools using an image based insertion-shaft detection algorithm and probabilistic models. We apply our proposed approach in both a structured environment as well as an unstructured environment and measure to demonstrate the efficacy of our methods.