# Single-view robot pose and joint angle estimation via render & compare

arXiv: https://arxiv.org/abs/2104.09359 · HTML: https://mlsanity.com/paper/2104.09359/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 63 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [CosyPose: Consistent multi-view multi-object 6D pose estimation](https://mlsanity.com/paper/2008.08465/) (2020)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [BOP: Benchmark for 6D Object Pose Estimation](https://mlsanity.com/paper/1808.08319/) (2018)
- [SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again](https://mlsanity.com/paper/1711.10006/) (2017)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth](https://mlsanity.com/paper/1703.10896/) (2017)
- [Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation](https://mlsanity.com/paper/1908.07433/) (2019)
- [Disentangling Monocular 3D Object Detection](https://mlsanity.com/paper/1905.12365/) (2019)
- [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017)
- [BOP Challenge 2020 on 6D Object Localization](https://mlsanity.com/paper/2009.07378/) (2020)
- [DPOD: 6D Pose Object Detector and Refiner](https://mlsanity.com/paper/1902.11020/) (2019)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Category-Level Articulated Object Pose Estimation](https://mlsanity.com/paper/1912.11913/) (2019)
- [HybridPose: 6D Object Pose Estimation under Hybrid Representations](https://mlsanity.com/paper/2001.01869/) (2020)
- [Segmentation-driven 6D Object Pose Estimation](https://mlsanity.com/paper/1812.02541/) (2018)
- [6-DoF Object Pose from Semantic Keypoints](https://mlsanity.com/paper/1703.04670/) (2017)
- [Deep Part Induction from Articulated Object Pairs](https://mlsanity.com/paper/1809.07417/) (2018)
- [Deep Model-Based 6D Pose Refinement in RGB](https://mlsanity.com/paper/1810.03065/) (2018)
- [Camera-to-Robot Pose Estimation from a Single Image](https://mlsanity.com/paper/1911.09231/) (2019)
- [Generalized Feedback Loop for Joint Hand-Object Pose Estimation](https://mlsanity.com/paper/1903.10883/) (2019)
- [Training a Feedback Loop for Hand Pose Estimation](https://mlsanity.com/paper/1609.09698/) (2016)
- [Sim2Real View Invariant Visual Servoing by Recurrent Control](https://mlsanity.com/paper/1712.07642/) (2017)
- [CRAVES: Controlling Robotic Arm with a Vision-based Economic System](https://mlsanity.com/paper/1812.00725/) (2018)

## Built on by (most cited first, 30 of 33)

- [AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction](https://mlsanity.com/paper/2207.12909/) (2022)
- [Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects](https://mlsanity.com/paper/2112.11347/) (2021)
- [Self-Supervised Category-Level Articulated Object Pose Estimation with Part-Level SE(3) Equivariance](https://mlsanity.com/paper/2302.14268/) (2023)
- [Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer](https://mlsanity.com/paper/2302.14332/) (2023)
- [TexPose: Neural Texture Learning for Self-Supervised 6D Object Pose Estimation](https://mlsanity.com/paper/2212.12902/) (2022)
- [Differentiable Robot Rendering](https://mlsanity.com/paper/2410.13851/) (2024)
- [EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration](https://mlsanity.com/paper/2305.01191/) (2023)
- [CtRNet-X: Camera-to-Robot Pose Estimation in Real-world Conditions Using a Single Camera](https://mlsanity.com/paper/2409.10441/) (2024)
- [RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation through Embedding Predictive Pre-Training](https://mlsanity.com/paper/2411.17662/) (2024)
- [Image-based Pose Estimation and Shape Reconstruction for Robot Manipulators and Soft, Continuum Robots via Differentiable Rendering](https://mlsanity.com/paper/2302.14039/) (2023)
- [Real-time Holistic Robot Pose Estimation with Unknown States](https://mlsanity.com/paper/2402.05655/) (2024)
- [Learning Eye-in-Hand Camera Calibration from a Single Image](https://mlsanity.com/paper/2111.01245/) (2021)
- [Pooling Revisited: Your Receptive Field is Suboptimal](https://mlsanity.com/paper/2205.15254/) (2022)
- [Semi-Perspective Decoupled Heatmaps for 3D Robot Pose Estimation from Depth Maps](https://mlsanity.com/paper/2207.02519/) (2022)
- [Advances in 4D Representation: Geometry, Motion, and Interaction](https://mlsanity.com/paper/2510.19255/) (2025)
- [Learning Markerless Robot-Depth Camera Calibration and End-Effector Pose Estimation](https://mlsanity.com/paper/2212.07567/) (2022)
- [ROPA: Synthetic Robot Pose Generation for RGB-D Bimanual Data Augmentation](https://mlsanity.com/paper/2509.19454/) (2025)
- [Efficient Surgical Robotic Instrument Pose Reconstruction in Real World Conditions Using Unified Feature Detection](https://mlsanity.com/paper/2510.03532/) (2025)
- [A Multi-body Tracking Framework - From Rigid Objects to Kinematic Structures](https://mlsanity.com/paper/2208.01502/) (2022)
- [A Distance-Geometric Method for Recovering Robot Joint Angles From an RGB Image](https://mlsanity.com/paper/2301.02051/) (2023)
- [MonoSE(3)-Diffusion: A Monocular SE(3) Diffusion Framework for Robust Camera-to-Robot Pose Estimation](https://mlsanity.com/paper/2510.10434/) (2025)
- [GISR: Geometric Initialization and Silhouette-based Refinement for Single-View Robot Pose and Configuration Estimation](https://mlsanity.com/paper/2405.04890/) (2024)
- [Foundation Feature-Driven Online End-Effector Pose Estimation: A Marker-Free and Learning-Free Approach](https://mlsanity.com/paper/2503.14051/) (2025)
- [ToolTipNet: A Segmentation-Driven Deep Learning Baseline for Surgical Instrument Tip Detection](https://mlsanity.com/paper/2504.09700/) (2025)
- [Fiducial Exoskeletons: Image-Centric Robot State Estimation](https://mlsanity.com/paper/2601.08034/) (2026)
- [SurgCalib: Gaussian Splatting-Based Hand-Eye Calibration for Robot-Assisted Minimally Invasive Surgery](https://mlsanity.com/paper/2603.08983/) (2026)
- [3D Pose Nowcasting: Forecast the Future to Improve the Present](https://mlsanity.com/paper/2308.12914/) (2023)
- [Learning High-Fidelity Robot Self-Model with Articulated 3D Gaussian Splatting](https://mlsanity.com/paper/2503.05398/) (2025)
- [ARC-Calib: Autonomous Markerless Camera-to-Robot Calibration via Exploratory Robot Motions](https://mlsanity.com/paper/2503.14701/) (2025)
- [Latent Representations for Visual Proprioception in Inexpensive Robots](https://mlsanity.com/paper/2504.14634/) (2025)

## Abstract (arXiv)

We introduce RoboPose, a method to estimate the joint angles and the 6D camera-to-robot pose of a known articulated robot from a single RGB image. This is an important problem to grant mobile and itinerant autonomous systems the ability to interact with other robots using only visual information in non-instrumented environments, especially in the context of collaborative robotics. It is also challenging because robots have many degrees of freedom and an infinite space of possible configurations that often result in self-occlusions and depth ambiguities when imaged by a single camera. The contributions of this work are three-fold. First, we introduce a new render & compare approach for estimating the 6D pose and joint angles of an articulated robot that can be trained from synthetic data, generalizes to new unseen robot configurations at test time, and can be applied to a variety of robots. Second, we experimentally demonstrate the importance of the robot parametrization for the iterative pose updates and design a parametrization strategy that is independent of the robot structure. Finally, we show experimental results on existing benchmark datasets for four different robots and demonstrate that our method significantly outperforms the state of the art. Code and pre-trained models are available on the project webpage https://www.di.ens.fr/willow/research/robopose/.