mlsanityData to 7 Oct 2026

Telling the What while Pointing to the Where: Multimodal Queries for Image Retrieval

arXiv 2102.04980 ↗First posted 9 Feb 2021cs.CV cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Most existing image retrieval systems use text queries as a way for the user to express what they are looking for. However, fine-grained image retrieval often requires the ability to also express where in the image the content they are looking for is. The text modality can only cumbersomely express such localization preferences, whereas pointing is a more natural fit. In this paper, we propose an image retrieval setup with a new form of multimodal queries, where the user simultaneously uses both spoken natural language (the what) and mouse traces over an empty canvas (the where) to express the characteristics of the desired target image. We then describe simple modifications to an existing image retrieval model, enabling it to operate in this setup. Qualitative and quantitative experiments show that our model effectively takes this spatial guidance into account, and provides significantly more accurate retrieval results compared to text-only equivalent systems.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 54 of its 86 references that are in the map. First posted on arXiv 9 Feb 2021.

Builds on

54 of 86 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  12. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
38 more
  1. 2017Fine-tuning CNN Image Retrieval with No Human Annotationcited by 497
  2. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  3. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  4. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  5. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  6. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  7. 2016Large-Scale Image Retrieval with Attentive Deep Local Featurescited by 238
  8. 2018Composing Text and Image for Image Retrieval - An Empirical Odysseycited by 234
  9. 2019Connecting Vision and Language with Localized Narrativescited by 225
  10. 2016Speed/accuracy trade-offs for modern convolutional object detectorscited by 198
  11. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197
  12. 2014Learning Fine-grained Image Similarity with Deep Rankingcited by 189
  13. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  14. 2017Scene Graph Generation from Objects, Phrases and Region Captionscited by 154
  15. 2016Deep Image Retrieval: Learning global representations for image searchcited by 147
  16. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  17. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  18. 2020Learning the Best Pooling Strategy for Visual Semantic Embeddingcited by 136
  19. 2021Similarity Reasoning and Filtration for Image-Text Matchingcited by 121
  20. 2020IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrievalcited by 115
  21. 2016Dual Attention Networks for Multimodal Reasoning and Matchingcited by 113
  22. 2019CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrievalcited by 110
  23. 2018Dialog-based Interactive Image Retrievalcited by 84
  24. 2017Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrievalcited by 68
  25. 2016SIFT Meets CNN: A Decade Survey of Instance Retrievalcited by 66
  26. 2019Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrievalcited by 66
  27. 2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 63
  28. 2020Fine-grained Visual Textual Alignment for Cross-Modal Retrieval using Transformer Encoderscited by 47
  29. 2020Universal Weighting Metric Learning for Cross-Modal Matchingcited by 34
  30. 2020Adaptive Offline Quintuplet Loss for Image-Text Matchingcited by 26
  31. 2015Multimodal Convolutional Neural Networks for Matching Image and Sentencecited by 25
  32. 2017Recent Advance in Content-based Image Retrieval: A Literature Surveycited by 24
  33. 2020Transformer Reasoning Network for Image-Text Matching and Retrievalcited by 19
  34. 2016Linking Image and Text with 2-Way Netscited by 14
  35. 2020Learning to Represent Image and Text with Denotation Graphcited by 13
  36. 2020Structured Visual Search via Composition-aware Learningcited by 5
  37. 2017Region-Based Image Retrieval Revisitedcited by 4
  38. 2019Language Features Matter: Effective Language Representations for Vision-Language Taskscited by 4

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 2; 2024: 3; 2025: 2.

12 papers.

  1. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  2. 2022A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketchcited by 28
  3. 2023EDIS: Entity-Driven Image Search over Multimodal Web Contentcited by 27
  4. 2022FindIt: Generalized Localization with Natural Language Queriescited by 13
  5. 2022Answer-Me: Multi-Task Open-Vocabulary Visual Question Answeringcited by 12
  6. 2024MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agentscited by 11
  7. 2022Who are you referring to? Coreference resolution in image narrationscited by 7
  8. 2022UIGR: Unified Interactive Garment Retrievalcited by 6
  9. 2024A Survey of Multimodal Composite Editing and Retrievalcited by 5
  10. 2025Composite Sketch+Text Queries for Retrieving Objects with Elusive Names and Complex Interactionscited by 5
  11. 2025GENIUS: A Generative Framework for Universal Multimodal Searchcited by 5
  12. 2024CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Modelcited by 0