# Semantic keypoint-based pose estimation from single RGB frames

arXiv: https://arxiv.org/abs/2204.05864 · HTML: https://mlsanity.com/paper/2204.05864/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 98 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 30 of 50)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [Deeply-Supervised Nets](https://mlsanity.com/paper/1409.5185/) (2014)
- [Convolutional Pose Machines](https://mlsanity.com/paper/1602.00134/) (2016)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [CosyPose: Consistent multi-view multi-object 6D pose estimation](https://mlsanity.com/paper/2008.08465/) (2020)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [3D Bounding Box Estimation Using Deep Learning and Geometry](https://mlsanity.com/paper/1612.00496/) (2016)
- [BOP: Benchmark for 6D Object Pose Estimation](https://mlsanity.com/paper/1808.08319/) (2018)
- [SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again](https://mlsanity.com/paper/1711.10006/) (2017)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth](https://mlsanity.com/paper/1703.10896/) (2017)
- [Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation](https://mlsanity.com/paper/1908.07433/) (2019)
- [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017)
- [BOP Challenge 2020 on 6D Object Localization](https://mlsanity.com/paper/2009.07378/) (2020)
- [kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation](https://mlsanity.com/paper/1903.06684/) (2019)
- [EPOS: Estimating 6D Pose of Objects with Symmetries](https://mlsanity.com/paper/2004.00605/) (2020)
- [Human Pose Estimation with Iterative Error Feedback](https://mlsanity.com/paper/1507.06550/) (2015)
- [Instance-aware Semantic Segmentation via Multi-task Network Cascades](https://mlsanity.com/paper/1512.04412/) (2015)
- [Manopt, a Matlab toolbox for optimization on manifolds](https://mlsanity.com/paper/1308.5200/) (2013)
- [HybridPose: 6D Object Pose Estimation under Hybrid Representations](https://mlsanity.com/paper/2001.01869/) (2020)
- [Segmentation-driven 6D Object Pose Estimation](https://mlsanity.com/paper/1812.02541/) (2018)
- [Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views](https://mlsanity.com/paper/1505.05641/) (2015)
- [6-DoF Object Pose from Semantic Keypoints](https://mlsanity.com/paper/1703.04670/) (2017)
- [Annotating Object Instances with a Polygon-RNN](https://mlsanity.com/paper/1704.05548/) (2017)

## Built on by (most cited first, 7 of 7)

- [Object Pose Estimation with Statistical Guarantees: Conformal Keypoint Detection and Geometric Uncertainty Propagation](https://mlsanity.com/paper/2303.12246/) (2023)
- [Certifiable 3D Object Pose Estimation: Foundations, Learning Models, and Self-Training](https://mlsanity.com/paper/2206.11215/) (2022)
- [Optimal and Robust Category-level Perception: Object Pose and Shape Estimation from 2D and 3D Semantic Keypoints](https://mlsanity.com/paper/2206.12498/) (2022)
- [A Correct-and-Certify Approach to Self-Supervise Object Pose Estimators via Ensemble Self-Training](https://mlsanity.com/paper/2302.06019/) (2023)
- [Uncertainty Quantification for Visual Object Pose Estimation: S-Lemma Ellipsoidal Bounds](https://mlsanity.com/paper/2511.21666/) (2025)
- [Deterministic Object Pose Confidence Region Estimation](https://mlsanity.com/paper/2506.22720/) (2025)
- [Detecting Pose Estimation Failures via Keypoint Self-Consistency](https://mlsanity.com/paper/2608.03516/) (2026)

## Abstract (arXiv)

This paper presents an approach to estimating the continuous 6-DoF pose of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional network (convnet) with a deformable shape model. Unlike prior investigators, we are agnostic to whether the object is textured or textureless, as the convnet learns the optimal representation from the available training-image data. Furthermore, the approach can be applied to instance- and class-based pose recovery. Additionally, we accompany our main pipeline with a technique for semi-automatic data generation from unlabeled videos. This procedure allows us to train the learnable components of our method with minimal manual intervention in the labeling process. Empirically, we show that our approach can accurately recover the 6-DoF object pose for both instance- and class-based scenarios even against a cluttered background. We apply our approach both to several, existing, large-scale datasets - including PASCAL3D+, LineMOD-Occluded, YCB-Video, and TUD-Light - and, using our labeling pipeline, to a new dataset with novel object classes that we introduce here. Extensive empirical evaluations show that our approach is able to provide pose estimation results comparable to the state of the art.