# Real-time Holistic Robot Pose Estimation with Unknown States

arXiv: https://arxiv.org/abs/2402.05655 · HTML: https://mlsanity.com/paper/2402.05655/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 53 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Integral Human Pose Regression](https://mlsanity.com/paper/1711.08229/) (2017)
- [Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1907.11346/) (2019)
- [Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach](https://mlsanity.com/paper/1804.05172/) (2018)
- [Human Motion Generation: A Survey](https://mlsanity.com/paper/2307.10894/) (2023)
- [Camera-to-Robot Pose Estimation from a Single Image](https://mlsanity.com/paper/1911.09231/) (2019)
- [Learning Human-to-Robot Handovers from Point Clouds](https://mlsanity.com/paper/2303.17592/) (2023)
- [Single-view robot pose and joint angle estimation via render & compare](https://mlsanity.com/paper/2104.09359/) (2021)
- [Reactive Human-to-Robot Handovers of Arbitrary Objects](https://mlsanity.com/paper/2011.08961/) (2020)
- [Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer](https://mlsanity.com/paper/2302.14332/) (2023)
- [CRAVES: Controlling Robotic Arm with a Vision-based Economic System](https://mlsanity.com/paper/1812.00725/) (2018)
- [EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration](https://mlsanity.com/paper/2305.01191/) (2023)
- [Supervision via Competition: Robot Adversaries for Learning Tasks](https://mlsanity.com/paper/1610.01685/) (2016)
- [Robot Structure Prior Guided Temporal Attention for Camera-to-Robot Pose Estimation from Image Sequence](https://mlsanity.com/paper/2307.12106/) (2023)
- [Self-supervised Monocular Multi-robot Relative Localization with Efficient Deep Neural Networks](https://mlsanity.com/paper/2105.12797/) (2021)
- [External Camera-based Mobile Robot Pose Estimation for Collaborative Perception with Smart Edge Sensors](https://mlsanity.com/paper/2303.03797/) (2023)
- [Real-Time, Highly Accurate Robotic Grasp Detection using Fully Convolutional Neural Networks with High-Resolution Images](https://mlsanity.com/paper/1809.05828/) (2018)
- [Privacy-Preserving Pose Estimation for Human-Robot Interaction](https://mlsanity.com/paper/2011.07387/) (2020)

## Built on by (most cited first, 9 of 9)

- [RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation through Embedding Predictive Pre-Training](https://mlsanity.com/paper/2411.17662/) (2024)
- [ROSA: Harnessing Robot States for Vision-Language and Action Alignment](https://mlsanity.com/paper/2506.13679/) (2025)
- [MonoSE(3)-Diffusion: A Monocular SE(3) Diffusion Framework for Robust Camera-to-Robot Pose Estimation](https://mlsanity.com/paper/2510.10434/) (2025)
- [Fiducial Exoskeletons: Image-Centric Robot State Estimation](https://mlsanity.com/paper/2601.08034/) (2026)
- [Latent Representations for Visual Proprioception in Inexpensive Robots](https://mlsanity.com/paper/2504.14634/) (2025)
- [PoseDiff: A Unified Diffusion Model Bridging Robot Pose Estimation and Video-to-Action Control](https://mlsanity.com/paper/2509.24591/) (2025)
- [ROPES: Robotic Pose Estimation via Score-Based Causal Representation Learning](https://mlsanity.com/paper/2510.20884/) (2025)
- [RoboTAG: End-to-end Robot Configuration Estimation via Topological Alignment Graph](https://mlsanity.com/paper/2511.07717/) (2025)
- [Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Rendering](https://mlsanity.com/paper/2604.16388/) (2026)

## Abstract (arXiv)

Estimating robot pose from RGB images is a crucial problem in computer vision and robotics. While previous methods have achieved promising performance, most of them presume full knowledge of robot internal states, e.g. ground-truth robot joint angles. However, this assumption is not always valid in practical situations. In real-world applications such as multi-robot collaboration or human-robot interaction, the robot joint states might not be shared or could be unreliable. On the other hand, existing approaches that estimate robot pose without joint state priors suffer from heavy computation burdens and thus cannot support real-time applications. This work introduces an efficient framework for real-time robot pose estimation from RGB images without requiring known robot states. Our method estimates camera-to-robot rotation, robot state parameters, keypoint locations, and root depth, employing a neural network module for each task to facilitate learning and sim-to-real transfer. Notably, it achieves inference in a single feed-forward pass without iterative optimization. Our approach offers a 12-time speed increase with state-of-the-art accuracy, enabling real-time holistic robot pose estimation for the first time. Code and models are available at https://github.com/Oliverbansk/Holistic-Robot-Pose-Estimation.