# Understanding the Limitations of CNN-based Absolute Camera Pose Regression

arXiv: https://arxiv.org/abs/1903.07504 · HTML: https://mlsanity.com/paper/1903.07504/

## Key facts

- Cited by 177 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Mar 2019.

## Built on by (most cited first, 30 of 177)

- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors](https://mlsanity.com/paper/2103.17265/) (2021)
- [LENS: Localization enhanced by NeRF synthesis](https://mlsanity.com/paper/2110.06558/) (2021)
- [Learning Multi-Scene Absolute Pose Regression with Transformers](https://mlsanity.com/paper/2103.11468/) (2021)
- [Accelerated Coordinate Encoding: Learning to Relocalize in Minutes using RGB and Poses](https://mlsanity.com/paper/2305.14059/) (2023)
- [Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization](https://mlsanity.com/paper/2412.08376/) (2024)
- [Scene Coordinate Reconstruction: Posing of Image Collections via Incremental Learning of a Relocalizer](https://mlsanity.com/paper/2404.14351/) (2024)
- [ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformers](https://mlsanity.com/paper/2203.10157/) (2022)
- [RePOSE: Fast 6D Object Pose Refinement via Deep Texture Rendering](https://mlsanity.com/paper/2104.00633/) (2021)
- [On the Limits of Pseudo Ground Truth in Visual Camera Re-localisation](https://mlsanity.com/paper/2109.00524/) (2021)
- [MeshLoc: Mesh-Based Visual Localization](https://mlsanity.com/paper/2207.10762/) (2022)
- [Direct-PoseNet: Absolute Pose Regression with Photometric Consistency](https://mlsanity.com/paper/2104.04073/) (2021)
- [Is Geometry Enough for Matching in Visual Localization?](https://mlsanity.com/paper/2203.12979/) (2022)
- [GLACE: Global Local Accelerated Coordinate Encoding](https://mlsanity.com/paper/2406.04340/) (2024)
- [SOLD2: Self-supervised Occlusion-aware Line Description and Detection](https://mlsanity.com/paper/2104.03362/) (2021)
- [CROSSFIRE: Camera Relocalization On Self-Supervised Features from an Implicit Representation](https://mlsanity.com/paper/2303.04869/) (2023)
- [Data-efficient Large Scale Place Recognition with Graded Similarity Supervision](https://mlsanity.com/paper/2303.11739/) (2023)
- [Map-Relative Pose Regression for Visual Re-Localization](https://mlsanity.com/paper/2404.09884/) (2024)
- [Category-Level Metric Scale Object Shape and Pose Estimation](https://mlsanity.com/paper/2109.00326/) (2021)
- [Text2Pos: Text-to-Point-Cloud Cross-Modal Localization](https://mlsanity.com/paper/2203.15125/) (2022)
- [Visual Camera Re-Localization Using Graph Neural Networks and Relative Pose Supervision](https://mlsanity.com/paper/2104.02538/) (2021)
- [Neural Refinement for Absolute Pose Regression with Feature Synthesis](https://mlsanity.com/paper/2303.10087/) (2023)
- [Visual Localization via Few-Shot Scene Region Classification](https://mlsanity.com/paper/2208.06933/) (2022)
- [CoordiNet: uncertainty-aware pose regressor for reliable vehicle localization](https://mlsanity.com/paper/2103.10796/) (2021)
- [The NeRFect Match: Exploring NeRF Features for Visual Localization](https://mlsanity.com/paper/2403.09577/) (2024)
- [GS-CPR: Efficient Camera Pose Refinement via 3D Gaussian Splatting](https://mlsanity.com/paper/2408.11085/) (2024)
- [360Loc: A Dataset and Benchmark for Omnidirectional Visual Localization with Cross-device Queries](https://mlsanity.com/paper/2311.17389/) (2023)
- [LaLaLoc: Latent Layout Localisation in Dynamic, Unvisited Environments](https://mlsanity.com/paper/2104.09169/) (2021)
- [Large-scale Localization Datasets in Crowded Indoor Spaces](https://mlsanity.com/paper/2105.08941/) (2021)
- [NeRF-Loc: Visual Localization with Conditional Neural Radiance Field](https://mlsanity.com/paper/2304.07979/) (2023)

## Abstract (arXiv)

Visual localization is the task of accurate camera pose estimation in a known scene. It is a key problem in computer vision and robotics, with applications including self-driving cars, Structure-from-Motion, SLAM, and Mixed Reality. Traditionally, the localization problem has been tackled using 3D geometry. Recently, end-to-end approaches based on convolutional neural networks have become popular. These methods learn to directly regress the camera pose from an input image. However, they do not achieve the same level of pose accuracy as 3D structure-based methods. To understand this behavior, we develop a theoretical model for camera pose regression. We use our model to predict failure cases for pose regression techniques and verify our predictions through experiments. We furthermore use our model to show that pose regression is more closely related to pose approximation via image retrieval than to accurate pose estimation via 3D structure. A key result is that current approaches do not consistently outperform a handcrafted image retrieval baseline. This clearly shows that additional research is needed before pose regression algorithms are ready to compete with structure-based methods.