# Learning Analysis-by-Synthesis for 6D Pose Estimation in RGB-D Images

arXiv: https://arxiv.org/abs/1508.04546 · HTML: https://mlsanity.com/paper/1508.04546/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Aug 2015.

## Built on by (most cited first, 29 of 29)

- [Wide-Depth-Range 6D Object Pose Estimation in Space](https://mlsanity.com/paper/2104.00337/) (2021)
- [Occlusion-Aware Self-Supervised Monocular 6D Object Pose Estimation](https://mlsanity.com/paper/2203.10339/) (2022)
- [Perspective Flow Aggregation for Data-Limited 6D Object Pose Estimation](https://mlsanity.com/paper/2203.09836/) (2022)
- [Deep Learning on Monocular Object Pose Detection and Tracking: A Comprehensive Overview](https://mlsanity.com/paper/2105.14291/) (2021)
- [iCaps: Iterative Category-level Object Pose and Shape Estimation](https://mlsanity.com/paper/2201.00059/) (2021)
- [Neural Correspondence Field for Object Pose Estimation](https://mlsanity.com/paper/2208.00113/) (2022)
- [6D-ViT: Category-Level 6D Object Pose Estimation via Transformer-based Instance Representation Learning](https://mlsanity.com/paper/2110.04792/) (2021)
- [DemoGrasp: Few-Shot Learning for Robotic Grasping with Human Demonstration](https://mlsanity.com/paper/2112.02849/) (2021)
- [Sparse Steerable Convolutions: An Efficient Learning of SE(3)-Equivariant Features for Estimation and Tracking of Object Poses in 3D Space](https://mlsanity.com/paper/2111.07383/) (2021)
- [Shape-Constraint Recurrent Flow for 6D Object Pose Estimation](https://mlsanity.com/paper/2306.13266/) (2023)
- [PoET: Pose Estimation Transformer for Single-View, Multi-Object 6D Pose Estimation](https://mlsanity.com/paper/2211.14125/) (2022)
- [Towards Self-Supervised Category-Level Object Pose and Size Estimation](https://mlsanity.com/paper/2203.02884/) (2022)
- [Get a Grip: Reconstructing Hand-Object Stable Grasps in Egocentric Videos](https://mlsanity.com/paper/2312.15719/) (2023)
- [Pseudo Flow Consistency for Self-Supervised 6D Object Pose Estimation](https://mlsanity.com/paper/2308.10016/) (2023)
- [A Visual Navigation Perspective for Category-Level Object Pose Estimation](https://mlsanity.com/paper/2203.13572/) (2022)
- [Shadows Shed Light on 3D Objects](https://mlsanity.com/paper/2206.08990/) (2022)
- [Video based Object 6D Pose Estimation using Transformers](https://mlsanity.com/paper/2210.13540/) (2022)
- [3D Neural Embedding Likelihood: Probabilistic Inverse Graphics for Robust 6D Pose Estimation](https://mlsanity.com/paper/2302.03744/) (2023)
- [DSLR: Dynamic to Static LiDAR Scan Reconstruction Using Adversarially Trained Autoencoder](https://mlsanity.com/paper/2105.12774/) (2021)
- [WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects under Occlusion](https://mlsanity.com/paper/2403.19022/) (2024)
- [Recovering Parametric Scenes from Very Few Time-of-Flight Pixels](https://mlsanity.com/paper/2509.16132/) (2025)
- [Reconstructing Objects along Hand Interaction Timelines in Egocentric Video](https://mlsanity.com/paper/2512.07394/) (2025)
- [VideoPose: Estimating 6D object pose from videos](https://mlsanity.com/paper/2111.10677/) (2021)
- [ConvPoseCNN2: Prediction and Refinement of Dense 6D Object Poses](https://mlsanity.com/paper/2205.11124/) (2022)
- [RefPose: Leveraging Reference Geometric Correspondences for Accurate 6D Pose Estimation of Unseen Objects](https://mlsanity.com/paper/2505.10841/) (2025)
- [MorphGS: Morphology-Adaptive Articulated 3D Motion Transfer from Videos](https://mlsanity.com/paper/2601.02716/) (2026)
- [Canonical mapping as a general-purpose object descriptor for robotic manipulation](https://mlsanity.com/paper/2303.01331/) (2023)
- [TransPose: A Transformer-based 6D Object Pose Estimation Network with Depth Refinement](https://mlsanity.com/paper/2307.05561/) (2023)
- [Sparse Color-Code Net: Real-Time RGB-Based 6D Object Pose Estimation on Edge Devices](https://mlsanity.com/paper/2406.02977/) (2024)

## Abstract (arXiv)

Analysis-by-synthesis has been a successful approach for many tasks in computer vision, such as 6D pose estimation of an object in an RGB-D image which is the topic of this work. The idea is to compare the observation with the output of a forward process, such as a rendered image of the object of interest in a particular pose. Due to occlusion or complicated sensor noise, it can be difficult to perform this comparison in a meaningful way. We propose an approach that "learns to compare", while taking these difficulties into account. This is done by describing the posterior density of a particular object pose with a convolutional neural network (CNN) that compares an observed and rendered image. The network is trained with the maximum likelihood paradigm. We observe empirically that the CNN does not specialize to the geometry or appearance of specific objects, and it can be used with objects of vastly different shapes and appearances, and in different backgrounds. Compared to state-of-the-art, we demonstrate a significant improvement on two different datasets which include a total of eleven objects, cluttered background, and heavy occlusion.