# RNNPose: Recurrent 6-DoF Object Pose Refinement with Robust Correspondence Field Estimation and Pose Optimization

arXiv: https://arxiv.org/abs/2203.12870 · HTML: https://mlsanity.com/paper/2203.12870/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 62 references that are in the map.
- First posted on arXiv 24 Mar 2022.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Circle Loss: A Unified Perspective of Pair Similarity Optimization](https://mlsanity.com/paper/2002.10857/) (2020)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [BOP: Benchmark for 6D Object Pose Estimation](https://mlsanity.com/paper/1808.08319/) (2018)
- [SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again](https://mlsanity.com/paper/1711.10006/) (2017)
- [BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth](https://mlsanity.com/paper/1703.10896/) (2017)
- [Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation](https://mlsanity.com/paper/1908.07433/) (2019)
- [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017)
- [PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1911.04231/) (2019)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [DPOD: 6D Pose Object Detector and Refiner](https://mlsanity.com/paper/1902.11020/) (2019)
- [DeepV2D: Video to Depth with Differentiable Structure from Motion](https://mlsanity.com/paper/1812.04605/) (2018)
- [BA-Net: Dense Bundle Adjustment Network](https://mlsanity.com/paper/1806.04807/) (2018)

## Built on by (most cited first, 30 of 38)

- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation](https://mlsanity.com/paper/2303.12787/) (2023)
- [Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoning](https://mlsanity.com/paper/2205.00272/) (2022)
- [HS-Pose: Hybrid Scope Feature Extraction for Category-level Object Pose Estimation](https://mlsanity.com/paper/2303.15743/) (2023)
- [Instance-Adaptive and Geometric-Aware Keypoint Learning for Category-Level 6D Object Pose Estimation](https://mlsanity.com/paper/2403.19527/) (2024)
- [PATS: Patch Area Transportation with Subdivision for Local Feature Matching](https://mlsanity.com/paper/2303.07700/) (2023)
- [6D-Diff: A Keypoint Diffusion Framework for 6D Object Pose Estimation](https://mlsanity.com/paper/2401.00029/) (2023)
- [Shape-Constraint Recurrent Flow for 6D Object Pose Estimation](https://mlsanity.com/paper/2306.13266/) (2023)
- [CheckerPose: Progressive Dense Keypoint Localization for Object Pose Estimation with Graph Neural Network](https://mlsanity.com/paper/2303.16874/) (2023)
- [RGB-based Category-level Object Pose Estimation via Decoupled Metric Scale Recovery](https://mlsanity.com/paper/2309.10255/) (2023)
- [Do It Yourself: Learning Semantic Correspondence from Pseudo-Labels](https://mlsanity.com/paper/2506.05312/) (2025)
- [GS-Pose: Category-Level Object Pose Estimation via Geometric and Semantic Correspondence](https://mlsanity.com/paper/2311.13777/) (2023)
- [MATCHA:Towards Matching Anything](https://mlsanity.com/paper/2501.14945/) (2025)
- [MRC-Net: 6-DoF Pose Estimation with MultiScale Residual Correlation](https://mlsanity.com/paper/2403.08019/) (2024)
- [Pseudo Flow Consistency for Self-Supervised 6D Object Pose Estimation](https://mlsanity.com/paper/2308.10016/) (2023)
- [Depth-based 6DoF Object Pose Estimation using Swin Transformer](https://mlsanity.com/paper/2303.02133/) (2023)
- [Ambiguity-Aware Multi-Object Pose Optimization for Visually-Assisted Robot Manipulation](https://mlsanity.com/paper/2211.00960/) (2022)
- [Linear-Covariance Loss for End-to-End Learning of 6D Pose Estimation](https://mlsanity.com/paper/2303.11516/) (2023)
- [ETO:Efficient Transformer-based Local Feature Matching by Organizing Multiple Homography Hypotheses](https://mlsanity.com/paper/2410.22733/) (2024)
- [Advancing 6-DoF Instrument Pose Estimation in Variable X-Ray Imaging Geometries](https://mlsanity.com/paper/2405.11677/) (2024)
- [QORT-Former: Query-optimized Real-time Transformer for Understanding Two Hands Manipulating Objects](https://mlsanity.com/paper/2502.19769/) (2025)
- [Learning Implicit Probability Distribution Functions for Symmetric Orientation Estimation from RGB Images Without Pose Labels](https://mlsanity.com/paper/2211.11394/) (2022)
- [Manifold-Aware Self-Training for Unsupervised Domain Adaptation on Regressing 6D Object Pose](https://mlsanity.com/paper/2305.10808/) (2023)
- [For A More Comprehensive Evaluation of 6DoF Object Pose Tracking](https://mlsanity.com/paper/2309.07796/) (2023)
- [DeepRM: Deep Recurrent Matching for 6D Pose Refinement](https://mlsanity.com/paper/2205.14474/) (2022)
- [PixTrack: Precise 6DoF Object Pose Tracking using NeRF Templates and Feature-metric Alignment](https://mlsanity.com/paper/2209.03910/) (2022)
- [MV-ROPE: Multi-view Constraints for Robust Category-level Object Pose and Size Estimation](https://mlsanity.com/paper/2308.08856/) (2023)
- [Leveraging Positional Encoding for Robust Multi-Reference-Based Object 6D Pose Estimation](https://mlsanity.com/paper/2401.16284/) (2024)
- [A Flexible-Frame-Rate Vision-Aided Inertial Object Tracking System for Mobile Devices](https://mlsanity.com/paper/2210.12476/) (2022)
- [Towards Human-Level 3D Relative Pose Estimation: Generalizable, Training-Free, with Single Reference](https://mlsanity.com/paper/2406.18453/) (2024)

## Abstract (arXiv)

6-DoF object pose estimation from a monocular image is challenging, and a post-refinement procedure is generally needed for high-precision estimation. In this paper, we propose a framework based on a recurrent neural network (RNN) for object pose refinement, which is robust to erroneous initial poses and occlusions. During the recurrent iterations, object pose refinement is formulated as a non-linear least squares problem based on the estimated correspondence field (between a rendered image and the observed image). The problem is then solved by a differentiable Levenberg-Marquardt (LM) algorithm enabling end-to-end training. The correspondence field estimation and pose refinement are conducted alternatively in each iteration to recover the object poses. Furthermore, to improve the robustness to occlusion, we introduce a consistency-check mechanism based on the learned descriptors of the 3D model and observed 2D images, which downweights the unreliable correspondences during pose optimization. Extensive experiments on LINEMOD, Occlusion-LINEMOD, and YCB-Video datasets validate the effectiveness of our method and demonstrate state-of-the-art performance.