# EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation

arXiv: https://arxiv.org/abs/2203.13254 · HTML: https://mlsanity.com/paper/2203.13254/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 52 references that are in the map.
- First posted on arXiv 24 Mar 2022.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [3D Bounding Box Estimation Using Deep Learning and Geometry](https://mlsanity.com/paper/1612.00496/) (2016)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [Is Pseudo-Lidar needed for Monocular 3D Object detection?](https://mlsanity.com/paper/2108.06417/) (2021)
- [BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth](https://mlsanity.com/paper/1703.10896/) (2017)
- [Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation](https://mlsanity.com/paper/1908.07433/) (2019)
- [Disentangling Monocular 3D Object Detection](https://mlsanity.com/paper/1905.12365/) (2019)
- [Probabilistic and Geometric Depth: Detecting Objects in Perspective](https://mlsanity.com/paper/2107.14160/) (2021)
- [DPOD: 6D Pose Object Detector and Refiner](https://mlsanity.com/paper/1902.11020/) (2019)
- [Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild](https://mlsanity.com/paper/1911.11130/) (2019)
- [RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving](https://mlsanity.com/paper/2001.03343/) (2020)
- [Learning Less is More - 6D Camera Localization via 3D Surface Regression](https://mlsanity.com/paper/1711.10228/) (2017)
- [Bounding Box Regression with Uncertainty for Accurate Object Detection](https://mlsanity.com/paper/1809.08545/) (2018)
- [HybridPose: 6D Object Pose Estimation under Hybrid Representations](https://mlsanity.com/paper/2001.01869/) (2020)
- [Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction](https://mlsanity.com/paper/1906.03631/) (2019)

## Built on by (most cited first, 1 of 1)

- [CalibBEV: LiDAR-Camera Calibration via BEV Alignment](https://mlsanity.com/paper/2608.02309/) (2026)

## Abstract (arXiv)

Locating 3D objects from a single RGB image via Perspective-n-Points (PnP) is a long-standing problem in computer vision. Driven by end-to-end deep learning, recent studies suggest interpreting PnP as a differentiable layer, so that 2D-3D point correspondences can be partly learned by backpropagating the gradient w.r.t. object pose. Yet, learning the entire set of unrestricted 2D-3D points from scratch fails to converge with existing approaches, since the deterministic pose is inherently non-differentiable. In this paper, we propose the EPro-PnP, a probabilistic PnP layer for general end-to-end pose estimation, which outputs a distribution of pose on the SE(3) manifold, essentially bringing categorical Softmax to the continuous domain. The 2D-3D coordinates and corresponding weights are treated as intermediate variables learned by minimizing the KL divergence between the predicted and target pose distribution. The underlying principle unifies the existing approaches and resembles the attention mechanism. EPro-PnP significantly outperforms competitive baselines, closing the gap between PnP-based method and the task-specific leaders on the LineMOD 6DoF pose estimation and nuScenes 3D object detection benchmarks.