# Refining 6D Object Pose Predictions using Abstract Render-and-Compare

arXiv: https://arxiv.org/abs/1910.03412 · HTML: https://mlsanity.com/paper/1910.03412/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2019.

## Built on by (most cited first, 8 of 8)

- [YOLOPose: Transformer-based Multi-Object 6D Pose Estimation using Keypoint Regression](https://mlsanity.com/paper/2205.02536/) (2022)
- [T6D-Direct: Transformers for Multi-Object 6D Pose Direct Regression](https://mlsanity.com/paper/2109.10948/) (2021)
- [PFRL: Pose-Free Reinforcement Learning for 6D Pose Estimation](https://mlsanity.com/paper/2102.12096/) (2021)
- [Supervised Training of Dense Object Nets using Optimal Descriptors for Industrial Robotic Applications](https://mlsanity.com/paper/2102.08096/) (2021)
- [SynPick: A Dataset for Dynamic Bin Picking Scene Understanding](https://mlsanity.com/paper/2107.04852/) (2021)
- [YOLOPose V2: Understanding and Improving Transformer-based 6D Pose Estimation](https://mlsanity.com/paper/2307.11550/) (2023)
- [ConvPoseCNN2: Prediction and Refinement of Dense 6D Object Poses](https://mlsanity.com/paper/2205.11124/) (2022)
- [Efficient Multi-Object Pose Estimation using Multi-Resolution Deformable Attention and Query Aggregation](https://mlsanity.com/paper/2312.08268/) (2023)

## Abstract (arXiv)

Robotic systems often require precise scene analysis capabilities, especially in unstructured, cluttered situations, as occurring in human-made environments. While current deep-learning based methods yield good estimates of object poses, they often struggle with large amounts of occlusion and do not take inter-object effects into account. Vision as inverse graphics is a promising concept for detailed scene analysis. A key element for this idea is a method for inferring scene parameter updates from the rasterized 2D scene. However, the rasterization process is notoriously difficult to invert, both due to the projection and occlusion process, but also due to secondary effects such as lighting or reflections. We propose to remove the latter from the process by mapping the rasterized image into an abstract feature space learned in a self-supervised way from pixel correspondences. Using only a light-weight inverse rendering module, this allows us to refine 6D object pose estimations in highly cluttered scenes by optimizing a simple pixel-wise difference in the abstract image representation. We evaluate our approach on the challenging YCB-Video dataset, where it yields large improvements and demonstrates a large basin of attraction towards the correct object poses.