# Learning Eye-in-Hand Camera Calibration from a Single Image

arXiv: https://arxiv.org/abs/2111.01245 · HTML: https://mlsanity.com/paper/2111.01245/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 50 references that are in the map.
- First posted on arXiv 1 Nov 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [CosyPose: Consistent multi-view multi-object 6D pose estimation](https://mlsanity.com/paper/2008.08465/) (2020)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [DPOD: 6D Pose Object Detector and Refiner](https://mlsanity.com/paper/1902.11020/) (2019)
- [Deep Spatial Autoencoders for Visuomotor Learning](https://mlsanity.com/paper/1509.06113/) (2015)
- [Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration](https://mlsanity.com/paper/2105.06411/) (2021)
- [Camera-to-Robot Pose Estimation from a Single Image](https://mlsanity.com/paper/1911.09231/) (2019)
- [Single-view robot pose and joint angle estimation via render & compare](https://mlsanity.com/paper/2104.09359/) (2021)
- [Crossing The Gap: A Deep Dive into Zero-Shot Sim-to-Real Transfer for Dynamics](https://mlsanity.com/paper/2008.06686/) (2020)
- [KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation](https://mlsanity.com/paper/2007.13960/) (2020)
- [Benchmarking Domain Randomisation for Visual Sim-to-Real Transfer](https://mlsanity.com/paper/2011.07112/) (2020)
- [Deep Learning a Grasp Function for Grasping under Gripper Pose Uncertainty](https://mlsanity.com/paper/1608.02239/) (2016)
- [Learning Pose Estimation for High-Precision Robotic Assembly Using Simulated Depth Images](https://mlsanity.com/paper/1809.10699/) (2018)
- [Fast robust peg-in-hole insertion with continuous visual servoing](https://mlsanity.com/paper/2011.06399/) (2020)
- [Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spaces](https://mlsanity.com/paper/2105.11283/) (2021)
- [PnP-Net: A hybrid Perspective-n-Point Network](https://mlsanity.com/paper/2003.04626/) (2020)

## Built on by (most cited first, 8 of 8)

- [Among Us: Adversarially Robust Collaborative Perception by Consensus](https://mlsanity.com/paper/2303.09495/) (2023)
- [EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration](https://mlsanity.com/paper/2305.01191/) (2023)
- [Learning Markerless Robot-Depth Camera Calibration and End-Effector Pose Estimation](https://mlsanity.com/paper/2212.07567/) (2022)
- [3D Foundation Models Enable Simultaneous Geometry and Pose Estimation of Grasped Objects](https://mlsanity.com/paper/2407.10331/) (2024)
- [Bi-Manual Joint Camera Calibration and Scene Representation](https://mlsanity.com/paper/2505.24819/) (2025)
- [Unifying Scene Representation and Hand-Eye Calibration with 3D Foundation Models](https://mlsanity.com/paper/2404.11683/) (2024)
- [Kalib: Easy Hand-Eye Calibration with Reference Point Tracking](https://mlsanity.com/paper/2408.10562/) (2024)
- [DRHeC: Differentiable Rendering for Hand-Eye Calibration with RGB-Based Gradients](https://mlsanity.com/paper/2609.36779/) (2026)

## Abstract (arXiv)

Eye-in-hand camera calibration is a fundamental and long-studied problem in robotics. We present a study on using learning-based methods for solving this problem online from a single RGB image, whilst training our models with entirely synthetic data. We study three main approaches: one direct regression model that directly predicts the extrinsic matrix from an image, one sparse correspondence model that regresses 2D keypoints and then uses PnP, and one dense correspondence model that uses regressed depth and segmentation maps to enable ICP pose estimation. In our experiments, we benchmark these methods against each other and against well-established classical methods, to find the surprising result that direct regression outperforms other approaches, and we perform noise-sensitivity analysis to gain further insights into these results.