mlsanityData to 7 Oct 2026

VisualSparta: An Embarrassingly Simple Approach to Large-scale Text-to-Image Search with Weighted Bag-of-words

arXiv 2101.00265 ↗First posted 1 Jan 2021cs.CV cs.CL cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Text-to-image retrieval is an essential task in cross-modal information retrieval, i.e., retrieving relevant images from a large and unlabelled dataset given textual queries. In this paper, we propose VisualSparta, a novel (Visual-text Sparse Transformer Matching) model that shows significant improvement in terms of both accuracy and efficiency. VisualSparta is capable of outperforming previous state-of-the-art scalable methods in MSCOCO and Flickr30K. We also show that it achieves substantial retrieving speed advantages, i.e., for a 1 million image index, VisualSparta using CPU gets ~391X speedup compared to CPU vector search and ~5.4X speedup compared to vector search with GPU acceleration. Experiments show that this speed advantage even gets bigger for larger datasets because VisualSparta can be efficiently implemented as an inverted index. To the best of our knowledge, VisualSparta is the first transformer-based text-to-image retrieval model that can achieve real-time searching for large-scale datasets, with significant accuracy improvement compared to previous state-of-the-art methods.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 27 of its 33 references that are in the map. First posted on arXiv 1 Jan 2021.

Builds on

27 of 33 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2017Billion-scale similarity search with GPUscited by 2,477
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  12. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
15 more
  1. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  2. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  3. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  4. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  5. 2016Dual Attention Networks for Multimodal Reasoning and Matchingcited by 113
  6. 2019CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrievalcited by 110
  7. 2019ConveRT: Efficient and Accurate Conversational Representations from Transformerscited by 95
  8. 2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
  9. 2017Learning Semantic Concepts and Order for Image and Sentence Matchingcited by 61
  10. 2020Consensus-Aware Visual-Semantic Embedding for Image-Text Matchingcited by 59
  11. 2020SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrievalcited by 50
  12. 2019Position Focused Attention Network for Image-Text Matchingcited by 45
  13. 2016Instance-aware Image and Sentence Matching with Selective Multimodal LSTMcited by 43
  14. 2018Finding beans in burgers: Deep semantic-visual embedding with localizationcited by 28
  15. 2020Expansion via Prediction of Importance with Contextualizationcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 4; 2024: 2; 2025: 1; 2026: 2.

14 papers.

Sort:

By area:

  1. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  2. 2021Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrievalcited by 27
  3. 2023Chatting Makes Perfect: Chat-based Image Retrievalcited by 25
  4. 2022Where Does the Performance Improvement Come From? -- A Reproducibility Concern about Image-Text Retrievalcited by 19
  5. 2024Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Modelscited by 19
  6. 2022OmDet: Large-scale vision-language multi-dataset pre-training with multimodal detection networkcited by 13
  7. 2023Towards Fast and Accurate Image-Text Retrieval with Self-Supervised Fine-Grained Alignmentcited by 8
  8. 2023DATE: Domain Adaptive Product Seeker for E-commercecited by 4
  9. 2022Do Lessons from Metric Learning Generalize to Image-Caption Retrieval?cited by 3
  10. 2024RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-trainingcited by 3
  11. 2025Sparse and Dense Retrievers Learn Better Together: Joint Sparse-Dense Optimization for Text-Image Retrievalcited by 3
  12. 2026UEmbed: Unified Sparse and Dense Multimodal Embeddingscited by 2
2 more
  1. 2023A Challenging Multimodal Video Summary: Simultaneously Extracting and Generating Keyframe-Caption Pairs from Videocited by 1
  2. 2026CIRCLED: A Multi-turn CIR Dataset with Consistent Dialogues across Domainscited by 0