# Differentiable Rendering for Pose Estimation in Proximity Operations

arXiv: https://arxiv.org/abs/2212.12668 · HTML: https://mlsanity.com/paper/2212.12668/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 39 references that are in the map.
- First posted on arXiv 24 Dec 2022.

## Builds on (most cited first, 13 of 13)

- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again](https://mlsanity.com/paper/1711.10006/) (2017)
- [BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth](https://mlsanity.com/paper/1703.10896/) (2017)
- [Differentiable Rendering: A Survey](https://mlsanity.com/paper/2006.12057/) (2020)
- [LatentFusion: End-to-End Differentiable Reconstruction and Rendering for Unseen Object Pose Estimation](https://mlsanity.com/paper/1912.00416/) (2019)
- [Pose Estimation for Non-Cooperative Spacecraft Rendezvous Using Convolutional Neural Networks](https://mlsanity.com/paper/1809.07238/) (2018)
- [Towards Robust Learning-Based Pose Estimation of Noncooperative Spacecraft](https://mlsanity.com/paper/1909.00392/) (2019)
- [Introduction to Camera Pose Estimation with Deep Learning](https://mlsanity.com/paper/1907.05272/) (2019)
- [Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs](https://mlsanity.com/paper/1307.0060/) (2013)
- [FPGA Hardware Acceleration for Feature-Based Relative Navigation Applications](https://mlsanity.com/paper/2210.09481/) (2022)
- [NaRPA: Navigation and Rendering Pipeline for Astronautics](https://mlsanity.com/paper/2211.01566/) (2022)
- [Analytical Derivatives for Differentiable Renderer: 3D Pose Estimation by Silhouette Consistency](https://mlsanity.com/paper/1906.07870/) (2019)

## Abstract (arXiv)

Differentiable rendering aims to compute the derivative of the image rendering function with respect to the rendering parameters. This paper presents a novel algorithm for 6-DoF pose estimation through gradient-based optimization using a differentiable rendering pipeline. We emphasize two key contributions: (1) instead of solving the conventional 2D to 3D correspondence problem and computing reprojection errors, images (rendered using the 3D model) are compared only in the 2D feature space via sparse 2D feature correspondences. (2) Instead of an analytical image formation model, we compute an approximate local gradient of the rendering process through online learning. The learning data consists of image features extracted from multi-viewpoint renders at small perturbations in the pose neighborhood. The gradients are propagated through the rendering pipeline for the 6-DoF pose estimation using nonlinear least squares. This gradient-based optimization regresses directly upon the pose parameters by aligning the 3D model to reproduce a reference image shape. Using representative experiments, we demonstrate the application of our approach to pose estimation in proximity operations.