# DOLG: Single-Stage Image Retrieval with Deep Orthogonal Fusion of Local and Global Features

arXiv: https://arxiv.org/abs/2108.02927 · HTML: https://mlsanity.com/paper/2108.02927/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 6 Aug 2021.

## Builds on (most cited first, 27 of 27)

- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [Fine-tuning CNN Image Retrieval with No Human Annotation](https://mlsanity.com/paper/1711.02512/) (2017)
- [Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval](https://mlsanity.com/paper/2004.01804/) (2020)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [Large-Scale Image Retrieval with Attentive Deep Local Features](https://mlsanity.com/paper/1612.06321/) (2016)
- [R2D2: Repeatable and Reliable Detector and Descriptor](https://mlsanity.com/paper/1906.06195/) (2019)
- [Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking](https://mlsanity.com/paper/1803.11285/) (2018)
- [Learning with Average Precision: Training Image Retrieval with a Listwise Loss](https://mlsanity.com/paper/1906.07589/) (2019)
- [Beyond triplet loss: a deep quadruplet network for person re-identification](https://mlsanity.com/paper/1704.01719/) (2017)
- [Particular object retrieval with integral max-pooling of CNN activations](https://mlsanity.com/paper/1511.05879/) (2015)
- [End-to-end Learning of Deep Visual Representations for Image Retrieval](https://mlsanity.com/paper/1610.07940/) (2016)
- [Deep Metric Learning with Angular Loss](https://mlsanity.com/paper/1708.01682/) (2017)
- [Aggregating Deep Convolutional Features for Image Retrieval](https://mlsanity.com/paper/1510.07493/) (2015)
- [CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples](https://mlsanity.com/paper/1604.02426/) (2016)
- [Repeatability Is Not Enough: Learning Affine Regions via Discriminability](https://mlsanity.com/paper/1711.06704/) (2017)
- [Learning and aggregating deep local descriptors for instance-level recognition](https://mlsanity.com/paper/2007.13172/) (2020)
- [Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations](https://mlsanity.com/paper/1611.05113/) (2016)
- [SOLAR: Second-Order Loss and Attention for Image Retrieval](https://mlsanity.com/paper/2001.08972/) (2020)
- [Local Descriptors Optimized for Average Precision](https://mlsanity.com/paper/1804.05312/) (2018)
- [Detect-to-Retrieve: Efficient Regional Aggregation for Image Search](https://mlsanity.com/paper/1812.01584/) (2018)
- [Local Features and Visual Words Emerge in Activations](https://mlsanity.com/paper/1905.06358/) (2019)
- [Recent Advance in Content-based Image Retrieval: A Literature Survey](https://mlsanity.com/paper/1706.06064/) (2017)

## Built on by (most cited first, 30 of 54)

- [Global Features are All You Need for Image Retrieval and Reranking](https://mlsanity.com/paper/2308.06954/) (2023)
- [Towards Universal Image Embeddings: A Large-Scale Dataset and Challenge for Generic Image Representations](https://mlsanity.com/paper/2309.01858/) (2023)
- [Are Local Features All You Need for Cross-Domain Visual Place Recognition?](https://mlsanity.com/paper/2304.05887/) (2023)
- [Cross-Domain Video Anomaly Detection without Target Domain Adaptation](https://mlsanity.com/paper/2212.07010/) (2022)
- [Boosting vision transformers for image retrieval](https://mlsanity.com/paper/2210.11909/) (2022)
- [IRGen: Generative Modeling for Image Retrieval](https://mlsanity.com/paper/2303.10126/) (2023)
- [AMES: Asymmetric and Memory-Efficient Similarity Estimation for Instance-level Retrieval](https://mlsanity.com/paper/2408.03282/) (2024)
- [Bag of Tricks and A Strong baseline for Image Copy Detection](https://mlsanity.com/paper/2111.08004/) (2021)
- [Graph Convolution Based Efficient Re-Ranking for Visual Retrieval](https://mlsanity.com/paper/2306.08792/) (2023)
- [Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency](https://mlsanity.com/paper/2506.10178/) (2025)
- [Not All Frame Features Are Equal: Video-to-4D Generation via Decoupling Dynamic-Static Features](https://mlsanity.com/paper/2502.08377/) (2025)
- [VoxDet: Voxel Learning for Novel Instance Detection](https://mlsanity.com/paper/2305.17220/) (2023)
- [A New Benchmark and Model for Challenging Image Manipulation Detection](https://mlsanity.com/paper/2311.14218/) (2023)
- [Image Search with Text Feedback by Additive Attention Compositional Learning](https://mlsanity.com/paper/2203.03809/) (2022)
- [MOFI: Learning Image Representations from Noisy Entity Annotated Images](https://mlsanity.com/paper/2306.07952/) (2023)
- [Let All be Whitened: Multi-teacher Distillation for Efficient Visual Retrieval](https://mlsanity.com/paper/2312.09716/) (2023)
- [DALG: Deep Attentive Local and Global Modeling for Image Retrieval](https://mlsanity.com/paper/2207.00287/) (2022)
- [Orthogonal Temporal Interpolation for Zero-Shot Video Recognition](https://mlsanity.com/paper/2308.06897/) (2023)
- [Optimizing CLIP Models for Image Retrieval with Maintained Joint-Embedding Alignment](https://mlsanity.com/paper/2409.01936/) (2024)
- [Object-Centric Open-Vocabulary Image-Retrieval with Aggregated Features](https://mlsanity.com/paper/2309.14999/) (2023)
- [LOCORE: Image Re-ranking with Long-Context Sequence Modeling](https://mlsanity.com/paper/2503.21772/) (2025)
- [Leveraging BEV Representation for 360-degree Visual Place Recognition](https://mlsanity.com/paper/2305.13814/) (2023)
- [Yes, we CANN: Constrained Approximate Nearest Neighbors for local feature-based visual localization](https://mlsanity.com/paper/2306.09012/) (2023)
- [Coarse-to-Fine: Learning Compact Discriminative Representation for Single-Stage Image Retrieval](https://mlsanity.com/paper/2308.04008/) (2023)
- [Efficient large-scale image retrieval with deep feature orthogonality and Hybrid-Swin-Transformers](https://mlsanity.com/paper/2110.03786/) (2021)
- [Dark Side Augmentation: Generating Diverse Night Examples for Metric Learning](https://mlsanity.com/paper/2309.16351/) (2023)
- [On Train-Test Class Overlap and Detection for Image Retrieval](https://mlsanity.com/paper/2404.01524/) (2024)
- [Global-to-Local or Local-to-Global? Enhancing Image Retrieval with Efficient Local Search and Effective Global Re-ranking](https://mlsanity.com/paper/2509.04351/) (2025)
- [Hybrid Transformer Based Feature Fusion for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2211.11066/) (2022)
- [CDFI: Cross Domain Feature Interaction for Robust Bronchi Lumen Detection](https://mlsanity.com/paper/2304.09115/) (2023)

## Abstract (arXiv)

Image Retrieval is a fundamental task of obtaining images similar to the query one from a database. A common image retrieval practice is to firstly retrieve candidate images via similarity search using global image features and then re-rank the candidates by leveraging their local features. Previous learning-based studies mainly focus on either global or local image representation learning to tackle the retrieval task. In this paper, we abandon the two-stage paradigm and seek to design an effective single-stage solution by integrating local and global information inside images into compact image representations. Specifically, we propose a Deep Orthogonal Local and Global (DOLG) information fusion framework for end-to-end image retrieval. It attentively extracts representative local information with multi-atrous convolutions and self-attention at first. Components orthogonal to the global image representation are then extracted from the local information. At last, the orthogonal components are concatenated with the global representation as a complementary, and then aggregation is performed to generate the final representation. The whole framework is end-to-end differentiable and can be trained with image-level labels. Extensive experimental results validate the effectiveness of our solution and show that our model achieves state-of-the-art image retrieval performances on Revisited Oxford and Paris datasets.