# Efficient Surgical Robotic Instrument Pose Reconstruction in Real World Conditions Using Unified Feature Detection

arXiv: https://arxiv.org/abs/2510.03532 · HTML: https://mlsanity.com/paper/2510.03532/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 3 Oct 2025.

## Builds on (most cited first, 17 of 17)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Sapiens: Foundation for Human Vision Models](https://mlsanity.com/paper/2408.12569/) (2024)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression](https://mlsanity.com/paper/1904.07399/) (2019)
- [ViTPose++: Vision Transformer for Generic Body Pose Estimation](https://mlsanity.com/paper/2212.04246/) (2022)
- [Camera-to-Robot Pose Estimation from a Single Image](https://mlsanity.com/paper/1911.09231/) (2019)
- [Deep Hough Transform for Semantic Line Detection](https://mlsanity.com/paper/2003.04676/) (2020)
- [Single-view robot pose and joint angle estimation via render & compare](https://mlsanity.com/paper/2104.09359/) (2021)
- [SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction](https://mlsanity.com/paper/2003.03472/) (2020)
- [Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer](https://mlsanity.com/paper/2302.14332/) (2023)
- [Robotic Tool Tracking under Partially Visible Kinematic Chain: A Unified Approach](https://mlsanity.com/paper/2102.06235/) (2021)
- [Pose Estimation for Robot Manipulators via Keypoint Optimization and Sim-to-Real Transfer](https://mlsanity.com/paper/2010.08054/) (2020)
- [Fast and Reliable Autonomous Surgical Debridement with Cable-Driven Robots Using a Two-Phase Calibration Procedure](https://mlsanity.com/paper/1709.06668/) (2017)
- [CtRNet-X: Camera-to-Robot Pose Estimation in Real-world Conditions Using a Single Camera](https://mlsanity.com/paper/2409.10441/) (2024)
- [Image-based Pose Estimation and Shape Reconstruction for Robot Manipulators and Soft, Continuum Robots via Differentiable Rendering](https://mlsanity.com/paper/2302.14039/) (2023)
- [Differentiable Rendering-based Pose Estimation for Surgical Robotic Instruments](https://mlsanity.com/paper/2503.05953/) (2025)
- [Robust Surgical Tool Tracking with Pixel-based Probabilities for Projected Geometric Primitives](https://mlsanity.com/paper/2403.04971/) (2024)

## Built on by (most cited first, 3 of 3)

- [SurgCalib: Gaussian Splatting-Based Hand-Eye Calibration for Robot-Assisted Minimally Invasive Surgery](https://mlsanity.com/paper/2603.08983/) (2026)
- [MarUco: A Markerless 6D Pose Estimation Framework for Closed-Loop Control of Surgical Continuum Manipulators](https://mlsanity.com/paper/2602.16365/) (2026)
- [Robust Surgical Robotic Instrument Tracking via Sequential Multi-Cue Fusion and Sim-to-Real Self-Training](https://mlsanity.com/paper/2610.05491/) (2026)

## Abstract (arXiv)

Accurate camera-to-robot calibration is essential for any vision-based robotic control system and especially critical in minimally invasive surgical robots, where instruments conduct precise micro-manipulations. However, MIS robots have long kinematic chains and partial visibility of their degrees of freedom in the camera, which introduces challenges for conventional camera-to-robot calibration methods that assume stiff robots with good visibility. Previous works have investigated both keypoint-based and rendering-based approaches to address this challenge in real-world conditions; however, they often struggle with consistent feature detection or have long inference times, neither of which are ideal for online robot control. In this work, we propose a novel framework that unifies the detection of geometric primitives (keypoints and shaft edges) through a shared encoding, enabling efficient pose estimation via projection geometry. This architecture detects both keypoints and edges in a single inference and is trained on large-scale synthetic data with projective labeling. This method is evaluated across both feature detection and pose estimation, with qualitative and quantitative results demonstrating fast performance and state-of-the-art accuracy in challenging surgical environments.