# Hydra: Marker-Free RGB-D Hand-Eye Calibration

arXiv: https://arxiv.org/abs/2504.20584 · HTML: https://mlsanity.com/paper/2504.20584/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 29 Apr 2025.

## Builds on (most cited first, 16 of 16)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [Robot Operating System 2: Design, Architecture, and Uses In The Wild](https://mlsanity.com/paper/2211.07752/) (2022)
- [Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Study](https://mlsanity.com/paper/1404.3785/) (2014)
- [Camera-to-Robot Pose Estimation from a Single Image](https://mlsanity.com/paper/1911.09231/) (2019)
- [A Purely Algebraic Justification of the Kabsch-Umeyama Algorithm](https://mlsanity.com/paper/1902.03138/) (2019)
- [Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer](https://mlsanity.com/paper/2302.14332/) (2023)
- [EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration](https://mlsanity.com/paper/2305.01191/) (2023)
- [CtRNet-X: Camera-to-Robot Pose Estimation in Real-world Conditions Using a Single Camera](https://mlsanity.com/paper/2409.10441/) (2024)
- [EasyHeC++: Fully Automatic Hand-Eye Calibration with Pretrained Image Models](https://mlsanity.com/paper/2410.09293/) (2024)
- [OpTaS: An Optimization-based Task Specification Library for Trajectory Optimization and Model Predictive Control](https://mlsanity.com/paper/2301.13512/) (2023)
- [Automatic Robot Hand-Eye Calibration Enabled by Learning-Based 3D Vision](https://mlsanity.com/paper/2311.01335/) (2023)
- [Redundancy-aware Action Spaces for Robot Learning](https://mlsanity.com/paper/2406.04144/) (2024)
- [LBR-Stack: ROS 2 and Python Integration of KUKA FRI for Med and IIWA Robots](https://mlsanity.com/paper/2311.12709/) (2023)
- [RegHEC: Hand-Eye Calibration via Simultaneous Multi-view Point Clouds Registration of Arbitrary Object](https://mlsanity.com/paper/2304.14092/) (2023)

## Built on by (most cited first, 1 of 1)

- [Localising under the drape: proprioception in the era of distributed surgical robotic system](https://mlsanity.com/paper/2510.23512/) (2025)

## Abstract (arXiv)

This work presents an RGB-D imaging-based approach to marker-free hand-eye calibration using a novel implementation of the iterative closest point (ICP) algorithm with a robust point-to-plane (PTP) objective formulated on a Lie algebra. Its applicability is demonstrated through comprehensive experiments using three well known serial manipulators and two RGB-D cameras. With only three randomly chosen robot configurations, our approach achieves approximately 90% successful calibrations, demonstrating 2-3x higher convergence rates to the global optimum compared to both marker-based and marker-free baselines. We also report 2 orders of magnitude faster convergence time (0.8 +/- 0.4 s) for 9 robot configurations over other marker-free methods. Our method exhibits significantly improved accuracy (5 mm in task space) over classical approaches (7 mm in task space) whilst being marker-free. The benchmarking dataset and code are open sourced under Apache 2.0 License, and a ROS 2 integration with robot abstraction is provided to facilitate deployment.