# RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation through Embedding Predictive Pre-Training

arXiv: https://arxiv.org/abs/2411.17662 · HTML: https://mlsanity.com/paper/2411.17662/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 20 of 20)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Masked Siamese Networks for Label-Efficient Learning](https://mlsanity.com/paper/2204.07141/) (2022)
- [FractalNet: Ultra-Deep Neural Networks without Residuals](https://mlsanity.com/paper/1605.07648/) (2016)
- [End-to-End Learnable Geometric Vision by Backpropagating PnP Optimization](https://mlsanity.com/paper/1909.06043/) (2019)
- [Camera-to-Robot Pose Estimation from a Single Image](https://mlsanity.com/paper/1911.09231/) (2019)
- [Learning Human-to-Robot Handovers from Point Clouds](https://mlsanity.com/paper/2303.17592/) (2023)
- [Single-view robot pose and joint angle estimation via render & compare](https://mlsanity.com/paper/2104.09359/) (2021)
- [Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer](https://mlsanity.com/paper/2302.14332/) (2023)
- [RoboKeyGen: Robot Pose and Joint Angles Estimation via Diffusion-based 3D Keypoint Generation](https://mlsanity.com/paper/2403.18259/) (2024)
- [Robot Structure Prior Guided Temporal Attention for Camera-to-Robot Pose Estimation from Image Sequence](https://mlsanity.com/paper/2307.12106/) (2023)
- [Real-time Holistic Robot Pose Estimation with Unknown States](https://mlsanity.com/paper/2402.05655/) (2024)
- [Self-supervised Monocular Multi-robot Relative Localization with Efficient Deep Neural Networks](https://mlsanity.com/paper/2105.12797/) (2021)
- [Semi-Perspective Decoupled Heatmaps for 3D Robot Pose Estimation from Depth Maps](https://mlsanity.com/paper/2207.02519/) (2022)

## Built on by (most cited first, 10 of 10)

- [3D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Tasks](https://mlsanity.com/paper/2505.05800/) (2025)
- [Is Forward Prediction Enough? Physical State Grounding for JEPA World Models](https://mlsanity.com/paper/2608.06799/) (2026)
- [MonoSE(3)-Diffusion: A Monocular SE(3) Diffusion Framework for Robust Camera-to-Robot Pose Estimation](https://mlsanity.com/paper/2510.10434/) (2025)
- [MiVLA: Towards Generalizable Vision-Language-Action Model with Human-Robot Mutual Imitation Pre-training](https://mlsanity.com/paper/2512.15411/) (2025)
- [Fiducial Exoskeletons: Image-Centric Robot State Estimation](https://mlsanity.com/paper/2601.08034/) (2026)
- [PoseDiff: A Unified Diffusion Model Bridging Robot Pose Estimation and Video-to-Action Control](https://mlsanity.com/paper/2509.24591/) (2025)
- [ROPES: Robotic Pose Estimation via Score-Based Causal Representation Learning](https://mlsanity.com/paper/2510.20884/) (2025)
- [RoboTAG: End-to-end Robot Configuration Estimation via Topological Alignment Graph](https://mlsanity.com/paper/2511.07717/) (2025)
- [AdaDINO: Context-Adaptive DINO-Distilled Vision Foundation Models for Efficient Open-Vocabulary Edge Inference](https://mlsanity.com/paper/2604.15622/) (2026)
- [Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Rendering](https://mlsanity.com/paper/2604.16388/) (2026)

## Abstract (arXiv)

Vision-based pose estimation of articulated robots with unknown joint angles has applications in collaborative robotics and human-robot interaction tasks. Current frameworks use neural network encoders to extract image features and downstream layers to predict joint angles and robot pose. While images of robots inherently contain rich information about the robot's physical structures, existing methods often fail to leverage it fully; therefore, limiting performance under occlusions and truncations. To address this, we introduce RoboPEPP, a method that fuses information about the robot's physical model into the encoder using a masking-based self-supervised embedding-predictive architecture. Specifically, we mask the robot's joints and pre-train an encoder-predictor model to infer the joints' embeddings from surrounding unmasked regions, enhancing the encoder's understanding of the robot's physical model. The pre-trained encoder-predictor pair, along with joint angle and keypoint prediction networks, is then fine-tuned for pose and joint angle estimation. Random masking of input during fine-tuning and keypoint filtering during evaluation further improves robustness. Our method, evaluated on several datasets, achieves the best results in robot pose and joint angle estimation while being the least sensitive to occlusions and requiring the lowest execution time.