# Investigating the Role of Image Retrieval for Visual Localization -- An exhaustive benchmark

arXiv: https://arxiv.org/abs/2205.15761 · HTML: https://mlsanity.com/paper/2205.15761/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 126 references that are in the map.
- First posted on arXiv 31 May 2022.

## Builds on (most cited first, 30 of 41)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [Fine-tuning CNN Image Retrieval with No Human Annotation](https://mlsanity.com/paper/1711.02512/) (2017)
- [D2-Net: A Trainable CNN for Joint Detection and Description of Local Features](https://mlsanity.com/paper/1905.03561/) (2019)
- [Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval](https://mlsanity.com/paper/2004.01804/) (2020)
- [Large-Scale Image Retrieval with Attentive Deep Local Features](https://mlsanity.com/paper/1612.06321/) (2016)
- [Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition](https://mlsanity.com/paper/2103.01486/) (2021)
- [Geometric Loss Functions for Camera Pose Regression with Deep Learning](https://mlsanity.com/paper/1704.00390/) (2017)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking](https://mlsanity.com/paper/1803.11285/) (2018)
- [Learning with Average Precision: Training Image Retrieval with a Listwise Loss](https://mlsanity.com/paper/1906.07589/) (2019)
- [Understanding the Limitations of CNN-based Absolute Camera Pose Regression](https://mlsanity.com/paper/1903.07504/) (2019)
- [Particular object retrieval with integral max-pooling of CNN activations](https://mlsanity.com/paper/1511.05879/) (2015)
- [Geometry-Aware Learning of Maps for Camera Localization](https://mlsanity.com/paper/1712.03342/) (2017)
- [Learning Less is More - 6D Camera Localization via 3D Surface Regression](https://mlsanity.com/paper/1711.10228/) (2017)
- [Image-based localization using LSTMs for structured feature correlation](https://mlsanity.com/paper/1611.07890/) (2016)
- [End-to-end Learning of Deep Visual Representations for Image Retrieval](https://mlsanity.com/paper/1610.07940/) (2016)
- [Hierarchical Scene Coordinate Classification and Regression for Visual Localization](https://mlsanity.com/paper/1909.06216/) (2019)
- [Revisiting IM2GPS in the Deep Learning Era](https://mlsanity.com/paper/1705.04838/) (2017)
- [Camera Relocalization by Computing Pairwise Relative Poses Using Convolutional Neural Network](https://mlsanity.com/paper/1707.09733/) (2017)
- [Expert Sample Consensus Applied to Camera Re-Localization](https://mlsanity.com/paper/1908.02484/) (2019)
- [Robust Image Retrieval-based Visual Localization using Kapture](https://mlsanity.com/paper/2007.13867/) (2020)
- [On the Limits of Pseudo Ground Truth in Visual Camera Re-localisation](https://mlsanity.com/paper/2109.00524/) (2021)
- [To Learn or Not to Learn: Visual Localization from Essential Matrices](https://mlsanity.com/paper/1908.01293/) (2019)
- [ArcFace: Additive Angular Margin Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.07698/) (2018)
- [Visual Instance Retrieval with Deep Convolutional Networks](https://mlsanity.com/paper/1412.6574/) (2014)
- [Cross-dimensional Weighting for Aggregated Deep Convolutional Features](https://mlsanity.com/paper/1512.04065/) (2015)
- [Benchmarking Image Retrieval for Visual Localization](https://mlsanity.com/paper/2011.11946/) (2020)

## Built on by (most cited first, 27 of 27)

- [Scene Coordinate Reconstruction: Posing of Image Collections via Incremental Learning of a Relocalizer](https://mlsanity.com/paper/2404.14351/) (2024)
- [GS-CPR: Efficient Camera Pose Refinement via 3D Gaussian Splatting](https://mlsanity.com/paper/2408.11085/) (2024)
- [The Unreasonable Effectiveness of Pre-Trained Features for Camera Pose Refinement](https://mlsanity.com/paper/2404.10438/) (2024)
- [SAIL-Recon: Large SfM by Augmenting Scene Regression with Localization](https://mlsanity.com/paper/2508.17972/) (2025)
- [Learning Cross-view Geo-localization Embeddings via Dynamic Weighted Decorrelation Regularization](https://mlsanity.com/paper/2211.05296/) (2022)
- [End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenon](https://mlsanity.com/paper/2309.16634/) (2023)
- [Multi-Level Embedding and Alignment Network with Consistency and Invariance Learning for Cross-View Geo-Localization](https://mlsanity.com/paper/2412.14819/) (2024)
- [General Place Recognition Survey: Towards the Real-world Autonomy Age](https://mlsanity.com/paper/2209.04497/) (2022)
- [Gaussian Splatting Feature Fields for Privacy-Preserving Visual Localization](https://mlsanity.com/paper/2507.23569/) (2025)
- [A Guide to Structureless Visual Localization](https://mlsanity.com/paper/2504.17636/) (2025)
- [Self-supervised Learning of Neural Implicit Feature Fields for Camera Pose Refinement](https://mlsanity.com/paper/2406.08463/) (2024)
- [LoD-Loc v2: Aerial Visual Localization over Low Level-of-Detail City Models using Explicit Silhouette Alignment](https://mlsanity.com/paper/2507.00659/) (2025)
- [Self-Supervised Learning for Place Representation Generalization across Appearance Changes](https://mlsanity.com/paper/2303.02370/) (2023)
- [Yes, we CANN: Constrained Approximate Nearest Neighbors for local feature-based visual localization](https://mlsanity.com/paper/2306.09012/) (2023)
- [Learning to Localize in Unseen Scenes with Relative Pose Regressors](https://mlsanity.com/paper/2303.02717/) (2023)
- [Dark Side Augmentation: Generating Diverse Night Examples for Metric Learning](https://mlsanity.com/paper/2309.16351/) (2023)
- [Can we make NeRF-based visual localization privacy-preserving?](https://mlsanity.com/paper/2508.18971/) (2025)
- [AIR-HLoc: Adaptive Retrieved Images Selection for Efficient Visual Localisation](https://mlsanity.com/paper/2403.18281/) (2024)
- [Weatherproofing Retrieval for Localization with Generative AI and Geometric Consistency](https://mlsanity.com/paper/2402.09237/) (2024)
- [YOWO: You Only Walk Once to Jointly Map An Indoor Scene and Register Ceiling-mounted Cameras](https://mlsanity.com/paper/2511.16521/) (2025)
- [Benchmarking Efficient & Effective Camera Pose Estimation Strategies for Novel View Synthesis](https://mlsanity.com/paper/2603.20428/) (2026)
- [3DoF Localization from a Single Image and an Object Map: the Flatlandia Problem and Dataset](https://mlsanity.com/paper/2304.06373/) (2023)
- [Towards Global Localization using Multi-Modal Object-Instance Re-Identification](https://mlsanity.com/paper/2409.12002/) (2024)
- [A-SCoRe: Attention-based Scene Coordinate Regression for wide-ranging scenarios](https://mlsanity.com/paper/2503.13982/) (2025)
- [Multiview Image-Based Localization](https://mlsanity.com/paper/2503.23577/) (2025)
- [LoD-Loc v3: Generalized Aerial Localization in Dense Cities using Instance Silhouette Alignment](https://mlsanity.com/paper/2603.19609/) (2026)
- [Hierarchical Visual Relocalization with Nearest View Synthesis from Feature Gaussian Splatting](https://mlsanity.com/paper/2603.29185/) (2026)

## Abstract (arXiv)

Visual localization, i.e., camera pose estimation in a known scene, is a core component of technologies such as autonomous driving and augmented reality. State-of-the-art localization approaches often rely on image retrieval techniques for one of two purposes: (1) provide an approximate pose estimate or (2) determine which parts of the scene are potentially visible in a given query image. It is common practice to use state-of-the-art image retrieval algorithms for both of them. These algorithms are often trained for the goal of retrieving the same landmark under a large range of viewpoint changes which often differs from the requirements of visual localization. In order to investigate the consequences for visual localization, this paper focuses on understanding the role of image retrieval for multiple visual localization paradigms. First, we introduce a novel benchmark setup and compare state-of-the-art retrieval representations on multiple datasets using localization performance as metric. Second, we investigate several definitions of "ground truth" for image retrieval. Using these definitions as upper bounds for the visual localization paradigms, we show that there is still sgnificant room for improvement. Third, using these tools and in-depth analysis, we show that retrieval performance on classical landmark retrieval or place recognition tasks correlates only for some but not all paradigms to localization performance. Finally, we analyze the effects of blur and dynamic scenes in the images. We conclude that there is a need for retrieval approaches specifically designed for localization paradigms. Our benchmark and evaluation protocols are available at https://github.com/naver/kapture-localization.