# Coupled Iterative Refinement for 6D Multi-Object Pose Estimation

arXiv: https://arxiv.org/abs/2204.12516 · HTML: https://mlsanity.com/paper/2204.12516/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 26 Apr 2022.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [CosyPose: Consistent multi-view multi-object 6D pose estimation](https://mlsanity.com/paper/2008.08465/) (2020)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [BOP: Benchmark for 6D Object Pose Estimation](https://mlsanity.com/paper/1808.08319/) (2018)
- [BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth](https://mlsanity.com/paper/1703.10896/) (2017)
- [Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation](https://mlsanity.com/paper/1908.07433/) (2019)
- [T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects](https://mlsanity.com/paper/1701.05498/) (2017)
- [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017)
- [PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1911.04231/) (2019)
- [EPOS: Estimating 6D Pose of Objects with Symmetries](https://mlsanity.com/paper/2004.00605/) (2020)
- [RAFT-3D: Scene Flow using Rigid-Motion Embeddings](https://mlsanity.com/paper/2012.00726/) (2020)
- [6-DoF Object Pose from Semantic Keypoints](https://mlsanity.com/paper/1703.04670/) (2017)
- [End-to-End Learnable Geometric Vision by Backpropagating PnP Optimization](https://mlsanity.com/paper/1909.06043/) (2019)
- [Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization](https://mlsanity.com/paper/2007.14628/) (2020)

## Built on by (most cited first, 30 of 48)

- [TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement](https://mlsanity.com/paper/2306.08637/) (2023)
- [MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare](https://mlsanity.com/paper/2212.06870/) (2022)
- [Infinite Photorealistic Worlds using Procedural Generation](https://mlsanity.com/paper/2306.09310/) (2023)
- [Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generation](https://mlsanity.com/paper/2406.11824/) (2024)
- [BOP Challenge 2022 on Detection, Segmentation and Pose Estimation of Specific Rigid Objects](https://mlsanity.com/paper/2302.13075/) (2023)
- [BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid Objects](https://mlsanity.com/paper/2403.09799/) (2024)
- [ReFit: Recurrent Fitting Network for 3D Human Recovery](https://mlsanity.com/paper/2308.11184/) (2023)
- [Deep Learning-Based Object Pose Estimation: A Comprehensive Survey](https://mlsanity.com/paper/2405.07801/) (2024)
- [GenFlow: Generalizable Recurrent Flow for 6D Pose Refinement of Novel Objects](https://mlsanity.com/paper/2403.11510/) (2024)
- [Challenges for Monocular 6D Object Pose Estimation in Robotics](https://mlsanity.com/paper/2307.12172/) (2023)
- [ZeroPose: CAD-Prompted Zero-shot Object 6D Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/2305.17934/) (2023)
- [Shape-Constraint Recurrent Flow for 6D Object Pose Estimation](https://mlsanity.com/paper/2306.13266/) (2023)
- [HiPose: Hierarchical Binary Surface Encoding and Correspondence Pruning for RGB-D 6DoF Object Pose Estimation](https://mlsanity.com/paper/2311.12588/) (2023)
- [CheckerPose: Progressive Dense Keypoint Localization for Object Pose Estimation with Graph Neural Network](https://mlsanity.com/paper/2303.16874/) (2023)
- [LaPose: Laplacian Mixture Shape Modeling for RGB-Based Category-Level Object Pose Estimation](https://mlsanity.com/paper/2409.15727/) (2024)
- [Co-op: Correspondence-based Novel Object Pose Estimation](https://mlsanity.com/paper/2503.17731/) (2025)
- [Rigidity-Aware Detection for 6D Object Pose Estimation](https://mlsanity.com/paper/2303.12396/) (2023)
- [OOD-CV-v2: An extended Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images](https://mlsanity.com/paper/2304.10266/) (2023)
- [Diff-DOPE: Differentiable Deep Object Pose Estimation](https://mlsanity.com/paper/2310.00463/) (2023)
- [MRC-Net: 6-DoF Pose Estimation with MultiScale Residual Correlation](https://mlsanity.com/paper/2403.08019/) (2024)
- [COPE: End-to-end trainable Constant Runtime Object Pose Estimation](https://mlsanity.com/paper/2208.08807/) (2022)
- [6D Object Pose Estimation from Approximate 3D Models for Orbital Robotics](https://mlsanity.com/paper/2303.13241/) (2023)
- [Pseudo Flow Consistency for Self-Supervised 6D Object Pose Estimation](https://mlsanity.com/paper/2308.10016/) (2023)
- [Ambiguity-Aware Multi-Object Pose Optimization for Visually-Assisted Robot Manipulation](https://mlsanity.com/paper/2211.00960/) (2022)
- [3D Neural Embedding Likelihood: Probabilistic Inverse Graphics for Robust 6D Pose Estimation](https://mlsanity.com/paper/2302.03744/) (2023)
- [Tracking by 3D Model Estimation of Unknown Objects in Videos](https://mlsanity.com/paper/2304.06419/) (2023)
- [Extending 6D Object Pose Estimators for Stereo Vision](https://mlsanity.com/paper/2402.05610/) (2024)
- [Physics-Based Rigid Body Object Tracking and Friction Filtering From RGB-D Videos](https://mlsanity.com/paper/2309.15703/) (2023)
- [SUP-NeRF: A Streamlined Unification of Pose Estimation and NeRF for Monocular 3D Object Reconstruction](https://mlsanity.com/paper/2403.15705/) (2024)
- [Open Challenges for Monocular Single-shot 6D Object Pose Estimation](https://mlsanity.com/paper/2302.11827/) (2023)

## Abstract (arXiv)

We address the task of 6D multi-object pose: given a set of known 3D objects and an RGB or RGB-D input image, we detect and estimate the 6D pose of each object. We propose a new approach to 6D object pose estimation which consists of an end-to-end differentiable architecture that makes use of geometric knowledge. Our approach iteratively refines both pose and correspondence in a tightly coupled manner, allowing us to dynamically remove outliers to improve accuracy. We use a novel differentiable layer to perform pose refinement by solving an optimization problem we refer to as Bidirectional Depth-Augmented Perspective-N-Point (BD-PnP). Our method achieves state-of-the-art accuracy on standard 6D Object Pose benchmarks. Code is available at https://github.com/princeton-vl/Coupled-Iterative-Refinement.