# Learning 2D-3D Correspondences To Solve The Blind Perspective-n-Point Problem

arXiv: https://arxiv.org/abs/2003.06752 · HTML: https://mlsanity.com/paper/2003.06752/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2020.

## Built on by (most cited first, 8 of 8)

- [Is Geometry Enough for Matching in Visual Localization?](https://mlsanity.com/paper/2203.12979/) (2022)
- [CorrI2P: Deep Image-to-Point Cloud Registration via Dense Correspondence](https://mlsanity.com/paper/2207.05483/) (2022)
- [FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators](https://mlsanity.com/paper/2310.03420/) (2023)
- [A Unified Framework for Implicit Sinkhorn Differentiation](https://mlsanity.com/paper/2205.06688/) (2022)
- [DGC-GNN: Leveraging Geometry and Color Cues for Visual Descriptor-Free 2D-3D Matching](https://mlsanity.com/paper/2306.12547/) (2023)
- [Sparse Semantic Map-Based Monocular Localization in Traffic Scenes Using Learned 2D-3D Point-Line Correspondences](https://mlsanity.com/paper/2210.04543/) (2022)
- [AEPnP: A Less-constrained EPnP Solver for Pose Estimation with Anisotropic Scaling](https://mlsanity.com/paper/2310.09982/) (2023)
- [DynamicGlue: Epipolar and Time-Informed Data Association in Dynamic Environments using Graph Neural Networks](https://mlsanity.com/paper/2403.11370/) (2024)

## Abstract (arXiv)

Conventional absolute camera pose via a Perspective-n-Point (PnP) solver often assumes that the correspondences between 2D image pixels and 3D points are given. When the correspondences between 2D and 3D points are not known a priori, the task becomes the much more challenging blind PnP problem. This paper proposes a deep CNN model which simultaneously solves for both the 6-DoF absolute camera pose and 2D--3D correspondences. Our model comprises three neural modules connected in sequence. First, a two-stream PointNet-inspired network is applied directly to both the 2D image keypoints and the 3D scene points in order to extract discriminative point-wise features harnessing both local and contextual information. Second, a global feature matching module is employed to estimate a matchability matrix among all 2D--3D pairs. Third, the obtained matchability matrix is fed into a classification module to disambiguate inlier matches. The entire network is trained end-to-end, followed by a robust model fitting (P3P-RANSAC) at test time only to recover the 6-DoF camera pose. Extensive tests on both real and simulated data have shown that our method substantially outperforms existing approaches, and is capable of processing thousands of points a second with the state-of-the-art accuracy.