# Indicative Image Retrieval: Turning Blackbox Learning into Grey

arXiv: https://arxiv.org/abs/2201.11898 · HTML: https://mlsanity.com/paper/2201.11898/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 67 references that are in the map.
- First posted on arXiv 28 Jan 2022.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Fine-tuning CNN Image Retrieval with No Human Annotation](https://mlsanity.com/paper/1711.02512/) (2017)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking](https://mlsanity.com/paper/1803.11285/) (2018)
- [Particular object retrieval with integral max-pooling of CNN activations](https://mlsanity.com/paper/1511.05879/) (2015)
- [Deep Image Retrieval: Learning global representations for image search](https://mlsanity.com/paper/1604.01325/) (2016)
- [Convolutional neural network architecture for geometric matching](https://mlsanity.com/paper/1703.05593/) (2017)
- [End-to-end Learning of Deep Visual Representations for Image Retrieval](https://mlsanity.com/paper/1610.07940/) (2016)
- [SIFT Meets CNN: A Decade Survey of Instance Retrieval](https://mlsanity.com/paper/1608.01807/) (2016)
- [Aggregating Deep Convolutional Features for Image Retrieval](https://mlsanity.com/paper/1510.07493/) (2015)
- [Hardness-Aware Deep Metric Learning](https://mlsanity.com/paper/1903.05503/) (2019)
- [Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations](https://mlsanity.com/paper/1611.05113/) (2016)
- [Visual Instance Retrieval with Deep Convolutional Networks](https://mlsanity.com/paper/1412.6574/) (2014)
- [Cross-dimensional Weighting for Aggregated Deep Convolutional Features](https://mlsanity.com/paper/1512.04065/) (2015)
- [Medical Image Retrieval using Deep Convolutional Neural Network](https://mlsanity.com/paper/1703.08472/) (2017)
- [Efficient Image Retrieval via Decoupling Diffusion into Online and Offline Processing](https://mlsanity.com/paper/1811.10907/) (2018)
- [Faster R-CNN Features for Instance Search](https://mlsanity.com/paper/1604.08893/) (2016)
- [Fast Spectral Ranking for Similarity Search](https://mlsanity.com/paper/1703.06935/) (2017)
- [Class-Weighted Convolutional Features for Visual Instance Search](https://mlsanity.com/paper/1707.02581/) (2017)
- [Siamese Network of Deep Fisher-Vector Descriptors for Image Retrieval](https://mlsanity.com/paper/1702.00338/) (2017)
- [Saliency Weighted Convolutional Features for Instance Search](https://mlsanity.com/paper/1711.10795/) (2017)
- [Learning Non-Metric Visual Similarity for Image Retrieval](https://mlsanity.com/paper/1709.01353/) (2017)
- [Packing and Padding: Coupled Multi-index for Accurate Image Retrieval](https://mlsanity.com/paper/1402.2681/) (2014)
- [Iterative Manifold Embedding Layer Learned by Incomplete Data for Large-scale Image Retrieval](https://mlsanity.com/paper/1707.09862/) (2017)

## Abstract (arXiv)

Deep learning became the game changer for image retrieval soon after it was introduced. It promotes the feature extraction (by representation learning) as the core of image retrieval, with the relevance/matching evaluation being degenerated into simple similarity metrics. In many applications, we need the matching evidence to be indicated rather than just have the ranked list (e.g., the locations of the target proteins/cells/lesions in medical images). It is like the matched words need to be highlighted in search engines. However, this is not easy to implement without explicit relevance/matching modeling. The deep representation learning models are not feasible because of their blackbox nature. In this paper, we revisit the importance of relevance/matching modeling in deep learning era with an indicative retrieval setting. The study shows that it is possible to skip the representation learning and model the matching evidence directly. By removing the dependency on the pre-trained models, it has avoided a lot of related issues (e.g., the domain gap between classification and retrieval, the detail-diffusion caused by convolution, and so on). More importantly, the study demonstrates that the matching can be explicitly modeled and backtracked later for generating the matching evidence indications. It can improve the explainability of deep inference. Our method obtains a best performance in literature on both Oxford-5k and Paris-6k, and sets a new record of 97.77% on Oxford-5k (97.81% on Paris-6k) without extracting any deep features.