# Automatic Robot Hand-Eye Calibration Enabled by Learning-Based 3D Vision

arXiv: https://arxiv.org/abs/2311.01335 · HTML: https://mlsanity.com/paper/2311.01335/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2023.

## Built on by (most cited first, 5 of 5)

- [Hydra: Marker-Free RGB-D Hand-Eye Calibration](https://mlsanity.com/paper/2504.20584/) (2025)
- [Augmenting cobots for sheet-metal SMEs with 3D object recognition and localisation](https://mlsanity.com/paper/2508.13964/) (2025)
- [Localising under the drape: proprioception in the era of distributed surgical robotic system](https://mlsanity.com/paper/2510.23512/) (2025)
- [Beyond the Vehicle: Cooperative Localization by Fusing Point Clouds for GPS-Challenged Urban Scenarios](https://mlsanity.com/paper/2602.03908/) (2026)
- [Continual Hand-Eye Calibration for Open-world Robotic Manipulation](https://mlsanity.com/paper/2604.15814/) (2026)

## Abstract (arXiv)

Hand-eye calibration, as a fundamental task in vision-based robotic systems, aims to estimate the transformation matrix between the coordinate frame of the camera and the robot flange. Most approaches to hand-eye calibration rely on external markers or human assistance. We proposed Look at Robot Base Once (LRBO), a novel methodology that addresses the hand-eye calibration problem without external calibration objects or human support, but with the robot base. Using point clouds of the robot base, a transformation matrix from the coordinate frame of the camera to the robot base is established as I=AXB. To this end, we exploit learning-based 3D detection and registration algorithms to estimate the location and orientation of the robot base. The robustness and accuracy of the method are quantified by ground-truth-based evaluation, and the accuracy result is compared with other 3D vision-based calibration methods. To assess the feasibility of our methodology, we carried out experiments utilizing a low-cost structured light scanner across varying joint configurations and groups of experiments. The proposed hand-eye calibration method achieved a translation deviation of 0.930 mm and a rotation deviation of 0.265 degrees according to the experimental results. Additionally, the 3D reconstruction experiments demonstrated a rotation error of 0.994 degrees and a position error of 1.697 mm. Moreover, our method offers the potential to be completed in 1 second, which is the fastest compared to other 3D hand-eye calibration methods. Code is released at github.com/leihui6/LRBO.