# Camera-to-Robot Pose Estimation from a Single Image

arXiv: https://arxiv.org/abs/1911.09231 · HTML: https://mlsanity.com/paper/1911.09231/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2019.

## Built on by (most cited first, 30 of 48)

- [Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects](https://mlsanity.com/paper/2112.11347/) (2021)
- [Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery](https://mlsanity.com/paper/2105.01047/) (2021)
- [Single-view robot pose and joint angle estimation via render & compare](https://mlsanity.com/paper/2104.09359/) (2021)
- [Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer](https://mlsanity.com/paper/2302.14332/) (2023)
- [Differentiable Robot Rendering](https://mlsanity.com/paper/2410.13851/) (2024)
- [SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo](https://mlsanity.com/paper/2106.16118/) (2021)
- [Generative Image as Action Models](https://mlsanity.com/paper/2407.07875/) (2024)
- [EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration](https://mlsanity.com/paper/2305.01191/) (2023)
- [Diff-DOPE: Differentiable Deep Object Pose Estimation](https://mlsanity.com/paper/2310.00463/) (2023)
- [CtRNet-X: Camera-to-Robot Pose Estimation in Real-world Conditions Using a Single Camera](https://mlsanity.com/paper/2409.10441/) (2024)
- [RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation through Embedding Predictive Pre-Training](https://mlsanity.com/paper/2411.17662/) (2024)
- [Image-based Pose Estimation and Shape Reconstruction for Robot Manipulators and Soft, Continuum Robots via Differentiable Rendering](https://mlsanity.com/paper/2302.14039/) (2023)
- [Real-time Holistic Robot Pose Estimation with Unknown States](https://mlsanity.com/paper/2402.05655/) (2024)
- [Learning Eye-in-Hand Camera Calibration from a Single Image](https://mlsanity.com/paper/2111.01245/) (2021)
- [Benchmarking Off-The-Shelf Solutions to Robotic Assembly Tasks](https://mlsanity.com/paper/2103.05140/) (2021)
- [Semi-Perspective Decoupled Heatmaps for 3D Robot Pose Estimation from Depth Maps](https://mlsanity.com/paper/2207.02519/) (2022)
- [NeRFtrinsic Four: An End-To-End Trainable NeRF Jointly Optimizing Diverse Intrinsic and Extrinsic Camera Parameters](https://mlsanity.com/paper/2303.09412/) (2023)
- [Structure from Action: Learning Interactions for Articulated Object 3D Structure Discovery](https://mlsanity.com/paper/2207.08997/) (2022)
- [External Camera-based Mobile Robot Pose Estimation for Collaborative Perception with Smart Edge Sensors](https://mlsanity.com/paper/2303.03797/) (2023)
- [Robust Surgical Tool Tracking with Pixel-based Probabilities for Projected Geometric Primitives](https://mlsanity.com/paper/2403.04971/) (2024)
- [MBAPose: Mask and Bounding-Box Aware Pose Estimation of Surgical Instruments with Photorealistic Domain Randomization](https://mlsanity.com/paper/2103.08105/) (2021)
- [CPnP: Consistent Pose Estimator for Perspective-n-Point Problem with Bias Elimination](https://mlsanity.com/paper/2209.05824/) (2022)
- [Learning Markerless Robot-Depth Camera Calibration and End-Effector Pose Estimation](https://mlsanity.com/paper/2212.07567/) (2022)
- [Efficient Surgical Robotic Instrument Pose Reconstruction in Real World Conditions Using Unified Feature Detection](https://mlsanity.com/paper/2510.03532/) (2025)
- [A Distance-Geometric Method for Recovering Robot Joint Angles From an RGB Image](https://mlsanity.com/paper/2301.02051/) (2023)
- [CGA-PoseNet: Camera Pose Regression via a 1D-Up Approach to Conformal Geometric Algebra](https://mlsanity.com/paper/2302.05211/) (2023)
- [MonoSE(3)-Diffusion: A Monocular SE(3) Diffusion Framework for Robust Camera-to-Robot Pose Estimation](https://mlsanity.com/paper/2510.10434/) (2025)
- [Incremental Learning for Animal Pose Estimation using RBF k-DPP](https://mlsanity.com/paper/2110.13598/) (2021)
- [GISR: Geometric Initialization and Silhouette-based Refinement for Single-View Robot Pose and Configuration Estimation](https://mlsanity.com/paper/2405.04890/) (2024)
- [Kalib: Easy Hand-Eye Calibration with Reference Point Tracking](https://mlsanity.com/paper/2408.10562/) (2024)

## Abstract (arXiv)

We present an approach for estimating the pose of an external camera with respect to a robot using a single RGB image of the robot. The image is processed by a deep neural network to detect 2D projections of keypoints (such as joints) associated with the robot. The network is trained entirely on simulated data using domain randomization to bridge the reality gap. Perspective-n-point (PnP) is then used to recover the camera extrinsics, assuming that the camera intrinsics and joint configuration of the robot manipulator are known. Unlike classic hand-eye calibration systems, our method does not require an off-line calibration step. Rather, it is capable of computing the camera extrinsics from a single frame, thus opening the possibility of on-line calibration. We show experimental results for three different robots and camera sensors, demonstrating that our approach is able to achieve accuracy with a single frame that is comparable to that of classic off-line hand-eye calibration using multiple frames. With additional frames from a static pose, accuracy improves even further. Code, datasets, and pretrained models for three widely-used robot manipulators are made available.