# End-to-end Learning of Deep Visual Representations for Image Retrieval

arXiv: https://arxiv.org/abs/1610.07940 · HTML: https://mlsanity.com/paper/1610.07940/

## Key facts

- Cited by 122 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2016.

## Built on by (most cited first, 30 of 122)

- [XCiT: Cross-Covariance Image Transformers](https://mlsanity.com/paper/2106.09681/) (2021)
- [OpenGAN: Open-Set Recognition via Open Data Generation](https://mlsanity.com/paper/2104.02939/) (2021)
- [TransVPR: Transformer-based place recognition with multi-level attention aggregation](https://mlsanity.com/paper/2201.02001/) (2022)
- [R2Former: Unified Retrieval and Reranking Transformer for Place Recognition](https://mlsanity.com/paper/2304.03410/) (2023)
- [Deep Visual Geo-localization Benchmark](https://mlsanity.com/paper/2204.03444/) (2022)
- [Deep Learning for Instance Retrieval: A Survey](https://mlsanity.com/paper/2101.11282/) (2021)
- [Instance-level Image Retrieval using Reranking Transformers](https://mlsanity.com/paper/2103.12236/) (2021)
- [Video Transformers: A Survey](https://mlsanity.com/paper/2201.05991/) (2022)
- [MeshLoc: Mesh-Based Visual Localization](https://mlsanity.com/paper/2207.10762/) (2022)
- [DOLG: Single-Stage Image Retrieval with Deep Orthogonal Fusion of Local and Global Features](https://mlsanity.com/paper/2108.02927/) (2021)
- [Recall@k Surrogate Loss with Large Batches and Similarity Mixup](https://mlsanity.com/paper/2108.11179/) (2021)
- [Correlation Verification for Image Retrieval](https://mlsanity.com/paper/2204.01458/) (2022)
- [Data-efficient Large Scale Place Recognition with Graded Similarity Supervision](https://mlsanity.com/paper/2303.11739/) (2023)
- [Global Features are All You Need for Image Retrieval and Reranking](https://mlsanity.com/paper/2308.06954/) (2023)
- [Viewpoint Invariant Dense Matching for Visual Geolocalization](https://mlsanity.com/paper/2109.09827/) (2021)
- [AdaFusion: Visual-LiDAR Fusion with Adaptive Weights for Place Recognition](https://mlsanity.com/paper/2111.11739/) (2021)
- [GS-CPR: Efficient Camera Pose Refinement via 3D Gaussian Splatting](https://mlsanity.com/paper/2408.11085/) (2024)
- [360Loc: A Dataset and Benchmark for Omnidirectional Visual Localization with Cross-device Queries](https://mlsanity.com/paper/2311.17389/) (2023)
- [ILIAS: Instance-Level Image retrieval At Scale](https://mlsanity.com/paper/2502.11748/) (2025)
- [Investigating the Role of Image Retrieval for Visual Localization -- An exhaustive benchmark](https://mlsanity.com/paper/2205.15761/) (2022)
- [LIP-Loc: LiDAR Image Pretraining for Cross-Modal Localization](https://mlsanity.com/paper/2312.16648/) (2023)
- [Noise-resistant Deep Metric Learning with Ranking-based Instance Selection](https://mlsanity.com/paper/2103.16047/) (2021)
- [UAVD4L: A Large-Scale Dataset for UAV 6-DoF Localization](https://mlsanity.com/paper/2401.05971/) (2024)
- [AddressCLIP: Empowering Vision-Language Models for City-wide Image Address Localization](https://mlsanity.com/paper/2407.08156/) (2024)
- [OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation](https://mlsanity.com/paper/2107.00249/) (2021)
- [Visual Localization using Imperfect 3D Models from the Internet](https://mlsanity.com/paper/2304.05947/) (2023)
- [Semantic Reinforced Attention Learning for Visual Place Recognition](https://mlsanity.com/paper/2108.08443/) (2021)
- [Robust and Decomposable Average Precision for Image Retrieval](https://mlsanity.com/paper/2110.01445/) (2021)
- [AMES: Asymmetric and Memory-Efficient Similarity Estimation for Instance-level Retrieval](https://mlsanity.com/paper/2408.03282/) (2024)
- [Contextual Similarity Aggregation with Self-attention for Visual Re-ranking](https://mlsanity.com/paper/2110.13430/) (2021)

## Abstract (arXiv)

While deep learning has become a key ingredient in the top performing methods for many computer vision tasks, it has failed so far to bring similar improvements to instance-level image retrieval. In this article, we argue that reasons for the underwhelming results of deep methods on image retrieval are threefold: i) noisy training data, ii) inappropriate deep architecture, and iii) suboptimal training procedure. We address all three issues. First, we leverage a large-scale but noisy landmark dataset and develop an automatic cleaning method that produces a suitable training set for deep retrieval. Second, we build on the recent R-MAC descriptor, show that it can be interpreted as a deep and differentiable architecture, and present improvements to enhance it. Last, we train this network with a siamese architecture that combines three streams with a triplet loss. At the end of the training process, the proposed architecture produces a global image representation in a single forward pass that is well suited for image retrieval. Extensive experiments show that our approach significantly outperforms previous retrieval approaches, including state-of-the-art methods based on costly local descriptor indexing and spatial verification. On Oxford 5k, Paris 6k and Holidays, we respectively report 94.7, 96.6, and 94.8 mean average precision. Our representations can also be heavily compressed using product quantization with little loss in accuracy. For additional material, please see www.xrce.xerox.com/Deep-Image-Retrieval.