mlsanityData to 7 Oct 2026

Training Vision Transformers for Image Retrieval

arXiv 2102.05644 ↗First posted 10 Feb 2021cs.CVcited by 90

Not on the map. Why?

Abstract (arXiv)

Transformers have shown outstanding results for natural language understanding and, more recently, for image classification. We here extend this work and propose a transformer-based approach for image retrieval: we adopt vision transformers for generating image descriptors and train the resulting model with a metric learning objective, which combines a contrastive loss with a differential entropy regularizer. Our results show consistent and significant improvements of transformers over convolution-based approaches. In particular, our method outperforms the state of the art on several public benchmarks for category-level retrieval, namely Stanford Online Product, In-Shop and CUB-200. Furthermore, our experiments on ROxford and RParis also show that, in comparable settings, transformers are competitive for particular object retrieval, especially in the regime of short vector representations and low-resolution images.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Builds on 35 of its 59 references that are in the map. First posted on arXiv 10 Feb 2021.

Builds on

35 of 59 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2017Non-local Neural Networkscited by 2,290
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  11. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  12. 2017Fine-tuning CNN Image Retrieval with No Human Annotationcited by 497
23 more
  1. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  2. 2019Multi-Similarity Loss with General Pair Weighting for Deep Metric Learningcited by 368
  3. 2020A Metric Learning Reality Checkcited by 317
  4. 2017Sampling Matters in Deep Embedding Learningcited by 285
  5. 2017No Fuss Distance Metric Learning using Proxiescited by 225
  6. 2018Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarkingcited by 189
  7. 2019Learning with Average Precision: Training Image Retrieval with a Listwise Losscited by 184
  8. 2019SoftTriple Loss: Deep Metric Learning Without Triplet Samplingcited by 163
  9. 2015Particular object retrieval with integral max-pooling of CNN activationscited by 160
  10. 2016Deep Image Retrieval: Learning global representations for image searchcited by 147
  11. 2019Cross-Batch Memory for Embedding Learningcited by 130
  12. 2018Spreading vectors for similarity searchcited by 114
  13. 2018Deep Metric Learning with Hierarchical Triplet Losscited by 112
  14. 2020ProxyNCA++: Revisiting and Revitalizing Proxy Neighborhood Component Analysiscited by 107
  15. 2020A unifying mutual information view of metric learning: cross-entropy vs. pairwise lossescited by 100
  16. 2018Attention-based Ensemble for Deep Metric Learningcited by 56
  17. 2018Improving Generalization via Scalable Neighborhood Component Analysiscited by 53
  18. 2019MIC: Mining Interclass Characteristics for Improved Metric Learningcited by 41
  19. 2018Deep Metric Learning with BIER: Boosting Independent Embeddings Robustlycited by 40
  20. 2017Learning Spread-out Local Feature Descriptorscited by 34
  21. 2019Metric Learning With HORDE: High-Order Regularizer for Deep Embeddingscited by 29
  22. 2019Hybrid-Attention based Decoupled Metric Learning for Zero-Shot Image Retrievalcited by 15
  23. 2021Investigating the Vision Transformer Model for Image Retrieval Taskscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 25; 2023: 27; 2024: 11; 2025: 2; 2026: 4.

90 papers.

Sort:

By area:

  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  3. 2021Multiscale Vision Transformerscited by 896
  4. 2021Going deeper with Image Transformerscited by 759
  5. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  6. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  7. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  8. 2022DeiT III: Revenge of the ViTcited by 417
  9. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  10. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  11. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  12. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
38 more
  1. 2022A Self-Supervised Descriptor for Image Copy Detectioncited by 175
  2. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  3. 2023AnyLoc: Towards Universal Visual Place Recognitioncited by 159
  4. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  5. 2022Hyperbolic Vision Transformers: Combining Improvements in Metric Learningcited by 109
  6. 2022TransVPR: Transformer-based place recognition with multi-level attention aggregationcited by 102
  7. 2021SiT: Self-supervised vIsion Transformercited by 99
  8. 2022Deep Visual Geo-localization Benchmarkcited by 83
  9. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 79
  10. 2021Feature Fusion Vision Transformer for Fine-Grained Visual Categorizationcited by 40
  11. 2022Residual Mixture of Expertscited by 39
  12. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  13. 2023Simple, Effective and General: A New Backbone for Cross-view Image Geo-localizationcited by 36
  14. 2022Handcrafted Histological Transformer (H2T): Unsupervised Representation of Whole Slide Imagescited by 32
  15. 2023SG-Former: Self-guided Transformer with Evolving Token Reallocationcited by 32
  16. 2021Transformer-Based Source-Free Domain Adaptationcited by 30
  17. 2022HIER: Metric Learning Beyond Class Labels via Hierarchical Regularizationcited by 26
  18. 2022Learning Sequential Descriptors for Sequence-based Visual Place Recognitioncited by 25
  19. 2023Chatting Makes Perfect: Chat-based Image Retrievalcited by 25
  20. 2022Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Trainingcited by 24
  21. 2021Person Re-Identification with a Locally Aware Transformercited by 19
  22. 2021Robust and Decomposable Average Precision for Image Retrievalcited by 19
  23. 2022Boosting vision transformers for image retrievalcited by 19
  24. 2023IRGen: Generative Modeling for Image Retrievalcited by 18
  25. 2021All the attention you need: Global-local, spatial-channel attention for image retrievalcited by 12
  26. 2021Scene-Adaptive Attention Network for Crowd Countingcited by 12
  27. 2023RoboLLM: Robotic Vision Tasks Grounded on Multimodal Large Language Modelscited by 12
  28. 2022BodyMap: Learning Full-Body Dense Correspondence Mapcited by 11
  29. 2023Graph Convolution Based Efficient Re-Ranking for Visual Retrievalcited by 11
  30. 2024A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationshipscited by 11
  31. 2024Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Featurescited by 11
  32. 2022A Large-scale Comprehensive Dataset and Copy-overlap Aware Evaluation Protocol for Segment-level Video Copy Detectioncited by 10
  33. 2023Referential communication in heterogeneous communities of pre-trained visual deep networkscited by 10
  34. 2023Efficient OCR for Building a Diverse Digital Historycited by 10
  35. 2023Optimization of Rank Losses for Image Retrievalcited by 10
  36. 2024Understanding Hyperbolic Metric Learning through Hard Negative Samplingcited by 10
  37. 2021Vision Transformer Hashing for Image Retrievalcited by 9
  38. 2023CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTscited by 9

50 most cited of 90 All 90 (JSON)