# Content-based Image Retrieval and the Semantic Gap in the Deep Learning Era

arXiv: https://arxiv.org/abs/2011.06490 · HTML: https://mlsanity.com/paper/2011.06490/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2020.

## Built on by (most cited first, 9 of 9)

- [Deep Learning for Instance Retrieval: A Survey](https://mlsanity.com/paper/2101.11282/) (2021)
- [Data Roaming and Quality Assessment for Composed Image Retrieval](https://mlsanity.com/paper/2303.09429/) (2023)
- [Active Learning via Classifier Impact and Greedy Selection for Interactive Image Retrieval](https://mlsanity.com/paper/2412.02310/) (2024)
- [Encoding Hierarchical Information in Neural Networks helps in Subpopulation Shift](https://mlsanity.com/paper/2112.10844/) (2021)
- [Dsfer-Net: A Deep Supervision and Feature Retrieval Network for Bitemporal Change Detection Using Modern Hopfield Networks](https://mlsanity.com/paper/2304.01101/) (2023)
- [StableSemantics: A Synthetic Language-Vision Dataset of Semantic Representations in Naturalistic Images](https://mlsanity.com/paper/2406.13735/) (2024)
- [Beyond Pixels: A Training-Free, Text-to-Text Framework for Remote Sensing Image Retrieval](https://mlsanity.com/paper/2512.10596/) (2025)
- [SearchAD: Large-Scale Rare Image Retrieval Dataset for Autonomous Driving](https://mlsanity.com/paper/2604.08008/) (2026)
- [Aggregating Neighbor Embedding Projection and Rank-Based Manifold Learning for Image Retrieval](https://mlsanity.com/paper/2609.01963/) (2026)

## Abstract (arXiv)

Content-based image retrieval has seen astonishing progress over the past decade, especially for the task of retrieving images of the same object that is depicted in the query image. This scenario is called instance or object retrieval and requires matching fine-grained visual patterns between images. Semantics, however, do not play a crucial role. This brings rise to the question: Do the recent advances in instance retrieval transfer to more generic image retrieval scenarios? To answer this question, we first provide a brief overview of the most relevant milestones of instance retrieval. We then apply them to a semantic image retrieval task and find that they perform inferior to much less sophisticated and more generic methods in a setting that requires image understanding. Following this, we review existing approaches to closing this so-called semantic gap by integrating prior world knowledge. We conclude that the key problem for the further advancement of semantic image retrieval lies in the lack of a standardized task definition and an appropriate benchmark dataset.