mlsanityData to 14 Oct 2026

FindIt: Generalized Localization with Natural Language Queries

arXiv 2203.17273 ↗First posted 31 Mar 2022cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

We propose FindIt, a simple and versatile framework that unifies a variety of visual grounding and localization tasks including referring expression comprehension, text-based localization, and object detection. Key to our architecture is an efficient multi-scale fusion module that unifies the disparate localization requirements across the tasks. In addition, we discover that a standard object detector is surprisingly effective in unifying these tasks without a need for task-specific design, losses, or pre-computed detections. Our end-to-end trainable framework responds flexibly and accurately to a wide range of referring expression, localization or detection queries for zero, one, or multiple objects. Jointly trained on these tasks, FindIt outperforms the state of the art on both referring expression and text-based localization, and shows competitive performance on object detection. Finally, FindIt generalizes better to out-of-distribution data and novel categories compared to strong single-task baselines. All of these are accomplished by a single, unified and efficient model. The code will be released.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 79 of its 99 references that are in the map. First posted on arXiv 31 Mar 2022.

Builds on

79 of 99 references.

References · 79

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2017Mask R-CNNcited by 7,277
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  9. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
38 more
  1. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  12. 2021Grounded Language-Image Pre-trainingcited by 1,156
  13. 2016Modeling Context in Referring Expressionscited by 1,063
  14. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  15. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  16. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  17. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 538
  18. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  19. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  20. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  21. 2016Visual Dialogcited by 377
  22. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  23. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  24. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  25. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  26. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 271
  27. 2016Segmentation from Natural Language Expressionscited by 257
  28. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 252
  29. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 251
  30. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  31. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  32. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 195
  33. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  34. 2018Zero-Shot Object Detectioncited by 189
  35. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  36. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  37. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 140
  38. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 138

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 10; 2024: 3.

13 papers.

Sort:

By area:

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  2. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 224
  3. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  4. 2023Described Object Detection: Liberating Object Detection with Flexible Expressionscited by 40
  5. 2023Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphscited by 33
  6. 2024Learning Visual Grounding from Generative Vision and Language Modelcited by 27
  7. 2023Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipationcited by 21
  8. 2023CAT-ViL: Co-Attention Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgerycited by 17
  9. 2024Hyperbolic Learning with Synthetic Captions for Open-World Detectioncited by 12
  10. 2023Open-Vocabulary Object Detection via Scene Graph Discoverycited by 7
  11. 2023Unified Open-Vocabulary Dense Visual Predictioncited by 4
  12. 2024How Important are Data Augmentations to Close the Domain Gap for Object Detection in Orbit?cited by 3
1 more
  1. 2023Language-Conditioned Observation Models for Visual Object Searchcited by 1