# MarUco: A Markerless 6D Pose Estimation Framework for Closed-Loop Control of Surgical Continuum Manipulators

arXiv: https://arxiv.org/abs/2602.16365 · HTML: https://mlsanity.com/paper/2602.16365/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 65 references that are in the map.
- First posted on arXiv 18 Feb 2026.

## Builds on (most cited first, 26 of 26)

- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023)
- [CosyPose: Consistent multi-view multi-object 6D pose estimation](https://mlsanity.com/paper/2008.08465/) (2020)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation](https://mlsanity.com/paper/1804.06534/) (2018)
- [Camera-to-Robot Pose Estimation from a Single Image](https://mlsanity.com/paper/1911.09231/) (2019)
- [SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics](https://mlsanity.com/paper/1909.05405/) (2019)
- [Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer](https://mlsanity.com/paper/2302.14332/) (2023)
- [ORBIT-Surgical: An Open-Simulation Framework for Learning Surgical Augmented Dexterity](https://mlsanity.com/paper/2404.16027/) (2024)
- [Intermittent Visual Servoing: Efficiently Learning Policies Robust to Instrument Changes for High-precision Surgical Manipulation](https://mlsanity.com/paper/2011.06163/) (2020)
- [CtRNet-X: Camera-to-Robot Pose Estimation in Real-world Conditions Using a Single Camera](https://mlsanity.com/paper/2409.10441/) (2024)
- [Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild](https://mlsanity.com/paper/2007.08939/) (2020)
- [Differentiable Rendering-based Pose Estimation for Surgical Robotic Instruments](https://mlsanity.com/paper/2503.05953/) (2025)
- [Shape-adaptive Hysteresis Compensation for Tendon-driven Continuum Manipulators](https://mlsanity.com/paper/2109.06907/) (2021)
- [Realistic Data Generation for 6D Pose Estimation of Surgical Instruments](https://mlsanity.com/paper/2406.07328/) (2024)
- [Non-linear Hysteresis Compensation of a Tendon-sheath-driven Robotic Manipulator using Motor Current](https://mlsanity.com/paper/2011.01817/) (2020)
- [MBAPose: Mask and Bounding-Box Aware Pose Estimation of Surgical Instruments with Photorealistic Domain Randomization](https://mlsanity.com/paper/2103.08105/) (2021)
- [MPPI-VS: Sampling-Based Model Predictive Control Strategy for Constrained Image-Based and Position-Based Visual Servoing](https://mlsanity.com/paper/2104.04925/) (2021)
- [Hysteresis Compensation of Flexible Continuum Manipulator using RGBD Sensing and Temporal Convolutional Network](https://mlsanity.com/paper/2402.11319/) (2024)
- [Efficient Surgical Robotic Instrument Pose Reconstruction in Real World Conditions Using Unified Feature Detection](https://mlsanity.com/paper/2510.03532/) (2025)
- [Towards Safe Control of Continuum Manipulator Using Shielded Multiagent Reinforcement Learning](https://mlsanity.com/paper/2106.07892/) (2021)
- [Deep Direct Visual Servoing of Tendon-Driven Continuum Robots](https://mlsanity.com/paper/2111.02580/) (2021)
- [SAM: Semi-Active Mechanism for Extensible Continuum Manipulator and Real-time Hysteresis Compensation Control Algorithm](https://mlsanity.com/paper/2406.18388/) (2024)
- [Configuration Tracking Control of a Multi-Segment Soft Robotic Arm Using a Cosserat Rod Model](https://mlsanity.com/paper/2210.00182/) (2022)
- [A Novel Feedforward Youla Parameterization Method for Avoiding Local Minima in Stereo Image Based Visual Servoing Control](https://mlsanity.com/paper/2506.10252/) (2025)

## Abstract (arXiv)

Flexible endoscopic continuum manipulators offer high dexterity and access to complex anatomy, but nonlinear hysteresis limits feedforward control accuracy. Closed-loop control can compensate for these errors but requires accurate six-degree-of-freedom (6D) end-effector pose feedback. We present MarUco, a markerless 6D pose estimation framework for closed-loop control using only stereo vision during operation. A photorealistic pseudo-rigid-body simulation pipeline generates large-scale annotated training data without manual labeling. A multifeature fusion network integrates masks, keypoints, heatmaps, and bounding boxes from both stereo views to estimate an initial pose, followed by a learned single-pass render-and-compare module that refines the pose without iterative optimization. Kinematics-free hand-eye calibration estimates camera-to-robot-base extrinsics, and self-supervised adaptation uses unlabeled real stereo pairs to mitigate sim-to-real pose error. Across 1,000 real samples, MarUco achieves translation and rotation errors of 0.78 ± 0.50 mm and 3.07 ± 1.25deg, respectively, with a total processing time of 52.4 ms per stereo pair. In closed-loop experiments over eight reference paths, MarUco achieves a mean terminal translation error of 1.8 mm, an 88% reduction relative to uncompensated open-loop control. To the best of our knowledge, this is the first markerless position-based visual servoing framework for continuum manipulators.