# Investigating the Vision Transformer Model for Image Retrieval Tasks

arXiv: https://arxiv.org/abs/2101.03771 · HTML: https://mlsanity.com/paper/2101.03771/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 45 references that are in the map.
- First posted on arXiv 11 Jan 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [Particular object retrieval with integral max-pooling of CNN activations](https://mlsanity.com/paper/1511.05879/) (2015)
- [Deep Image Retrieval: Learning global representations for image search](https://mlsanity.com/paper/1604.01325/) (2016)
- [SIFT Meets CNN: A Decade Survey of Instance Retrieval](https://mlsanity.com/paper/1608.01807/) (2016)
- [Visual Instance Retrieval with Deep Convolutional Networks](https://mlsanity.com/paper/1412.6574/) (2014)
- [Cross-dimensional Weighting for Aggregated Deep Convolutional Features](https://mlsanity.com/paper/1512.04065/) (2015)
- [Multi-scale Orderless Pooling of Deep Convolutional Activation Features](https://mlsanity.com/paper/1403.1840/) (2014)
- [Exploiting Local Features from Deep Networks for Image Retrieval](https://mlsanity.com/paper/1504.05133/) (2015)
- [L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks](https://mlsanity.com/paper/1802.09769/) (2018)
- [Descriptor Matching with Convolutional Neural Networks: a Comparison to SIFT](https://mlsanity.com/paper/1405.5769/) (2014)
- [Effective Image Retrieval via Multilinear Multi-index Fusion](https://mlsanity.com/paper/1709.09304/) (2017)

## Built on by (most cited first, 11 of 11)

- [Training Vision Transformers for Image Retrieval](https://mlsanity.com/paper/2102.05644/) (2021)
- [Deep Learning for Instance Retrieval: A Survey](https://mlsanity.com/paper/2101.11282/) (2021)
- [Defending Backdoor Attacks on Vision Transformer via Patch Processing](https://mlsanity.com/paper/2206.12381/) (2022)
- [Boosting vision transformers for image retrieval](https://mlsanity.com/paper/2210.11909/) (2022)
- [A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationships](https://mlsanity.com/paper/2408.15178/) (2024)
- [Vision Transformer Hashing for Image Retrieval](https://mlsanity.com/paper/2109.12564/) (2021)
- [6th Place Solution to Google Universal Image Embedding](https://mlsanity.com/paper/2210.09377/) (2022)
- [MATCHED: Multimodal Authorship-Attribution To Combat Human Trafficking in Escort-Advertisement Data](https://mlsanity.com/paper/2412.13794/) (2024)
- [Enhancing Landmark Detection in Cluttered Real-World Scenarios with Vision Transformers](https://mlsanity.com/paper/2308.13671/) (2023)
- [GvT: A Graph-based Vision Transformer with Talking-Heads Utilizing Sparsity, Trained from Scratch on Small Datasets](https://mlsanity.com/paper/2404.04924/) (2024)
- [SciceVPR: Stable Cross-Image Correlation Enhanced Model for Visual Place Recognition](https://mlsanity.com/paper/2502.20676/) (2025)

## Abstract (arXiv)

This paper introduces a plug-and-play descriptor that can be effectively adopted for image retrieval tasks without prior initialization or preparation. The description method utilizes the recently proposed Vision Transformer network while it does not require any training data to adjust parameters. In image retrieval tasks, the use of Handcrafted global and local descriptors has been very successfully replaced, over the last years, by the Convolutional Neural Networks (CNN)-based methods. However, the experimental evaluation conducted in this paper on several benchmarking datasets against 36 state-of-the-art descriptors from the literature demonstrates that a neural network that contains no convolutional layer, such as Vision Transformer, can shape a global descriptor and achieve competitive results. As fine-tuning is not required, the presented methodology's low complexity encourages adoption of the architecture as an image retrieval baseline model, replacing the traditional and well adopted CNN-based approaches and inaugurating a new era in image retrieval approaches.