# Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild

arXiv: https://arxiv.org/abs/2007.08939 · HTML: https://mlsanity.com/paper/2007.08939/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2020.

## Built on by (most cited first, 10 of 10)

- [Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Images](https://mlsanity.com/paper/2204.10776/) (2022)
- [RNNPose: Recurrent 6-DoF Object Pose Refinement with Robust Correspondence Field Estimation and Pose Optimization](https://mlsanity.com/paper/2203.12870/) (2022)
- [Cross-modal Learning for Image-Guided Point Cloud Shape Completion](https://mlsanity.com/paper/2209.09552/) (2022)
- [SPARC: Sparse Render-and-Compare for CAD model alignment in a single RGB image](https://mlsanity.com/paper/2210.01044/) (2022)
- [Extreme Two-View Geometry From Object Poses with Diffusion Models](https://mlsanity.com/paper/2402.02800/) (2024)
- [Sparse Multi-Object Render-and-Compare](https://mlsanity.com/paper/2310.11184/) (2023)
- [Category-Level Pose Retrieval with Contrastive Features Learnt with Occlusion Augmentation](https://mlsanity.com/paper/2208.06195/) (2022)
- [Personalized 3D Human Pose and Shape Refinement](https://mlsanity.com/paper/2403.11634/) (2024)
- [Generalizable Single-view Object Pose Estimation by Two-side Generating and Matching](https://mlsanity.com/paper/2411.15860/) (2024)
- [MarUco: A Markerless 6D Pose Estimation Framework for Closed-Loop Control of Surgical Continuum Manipulators](https://mlsanity.com/paper/2602.16365/) (2026)

## Abstract (arXiv)

We present a novel 3D pose refinement approach based on differentiable rendering for objects of arbitrary categories in the wild. In contrast to previous methods, we make two main contributions: First, instead of comparing real-world images and synthetic renderings in the RGB or mask space, we compare them in a feature space optimized for 3D pose refinement. Second, we introduce a novel differentiable renderer that learns to approximate the rasterization backward pass from data instead of relying on a hand-crafted algorithm. For this purpose, we predict deep cross-domain correspondences between RGB images and 3D model renderings in the form of what we call geometric correspondence fields. These correspondence fields serve as pixel-level gradients which are analytically propagated backward through the rendering pipeline to perform a gradient-based optimization directly on the 3D pose. In this way, we precisely align 3D models to objects in RGB images which results in significantly improved 3D pose estimates. We evaluate our approach on the challenging Pix3D dataset and achieve up to 55% relative improvement compared to state-of-the-art refinement methods in multiple metrics.