mlsanityData to 14 Oct 2026

Grounding of Textual Phrases in Images by Reconstruction

arXiv 1511.03745 ↗First posted 12 Nov 2015cs.CV cs.CL cs.LGcited by 112

Not on the map. Why?

Abstract (arXiv)

Grounding (i.e. localizing) arbitrary, free-form textual phrases in visual content is a challenging problem with many applications for human-computer interaction and image-text reference resolution. Few datasets provide the ground truth spatial localization of phrases, thus it is desirable to learn from data with no or little grounding supervision. We propose a novel approach which learns grounding by reconstructing a given phrase using an attention mechanism, which can be either latent or optimized directly. During training our approach encodes the phrase using a recurrent network language model and then learns to attend to the relevant image region in order to reconstruct the input phrase. At test time, the correct attention, i.e., the grounding, is evaluated. If grounding supervision is available it can be directly applied via a loss over the attention mechanism. We demonstrate the effectiveness of our approach on the Flickr 30k Entities and ReferItGame datasets with different levels of supervision, ranging from no supervision over partial supervision to full supervision. Our supervised variant improves by a large margin over the state-of-the-art on both datasets.

Key facts. Cited by 112 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 32; 2022: 35; 2023: 19; 2024: 12; 2025: 9; 2026: 5.

112 papers.

Sort:

By area:

  1. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  2. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 183
  3. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 126
  4. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 123
  5. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 120
  6. 2021Negative Sample Matters: A Renaissance of Metric Learning for Temporal Groundingcited by 90
  7. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 86
  8. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  9. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 69
  10. 2022Shifting More Attention to Visual Backbone: Query-modulated Refinement Networks for End-to-End Visual Groundingcited by 53
  11. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  12. 2024Toward Visual Grounding: A Surveycited by 52
38 more
  1. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 51
  2. 2021Towards General Purpose Vision Systemscited by 46
  3. 2021Visual Grounding with Transformerscited by 42
  4. 2022Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Groundingcited by 42
  5. 2021Injecting Semantic Concepts into End-to-End Image Captioningcited by 41
  6. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMscited by 38
  7. 2023CoTDet: Affordance Knowledge Prompting for Task Driven Object Detectioncited by 36
  8. 2021Relation-aware Instance Refinement for Weakly Supervised Visual Groundingcited by 35
  9. 2022Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Groundingcited by 35
  10. 2021Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Imagescited by 33
  11. 2021YouRefIt: Embodied Reference Understanding with Language and Gesturecited by 33
  12. 2023Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Groundingcited by 32
  13. 2022On Guiding Visual Attention with Language Specificationcited by 30
  14. 2021Linguistic Structures as Weak Supervision for Visual Scene Graph Generationcited by 29
  15. 2021Consensus Graph Representation Learning for Better Grounded Image Captioningcited by 28
  16. 2024OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modelingcited by 28
  17. 2022I2DFormer: Learning Image to Document Attention for Zero-Shot Image Classificationcited by 27
  18. 2023Shatter and Gather: Learning Referring Image Segmentation with Text Supervisioncited by 25
  19. 2022Improving Visual Grounding by Encouraging Consistent Gradient-based Explanationscited by 24
  20. 2021A Joint Network for Grasp Detection Conditioned on Natural Language Commandscited by 23
  21. 2022Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Groundingcited by 22
  22. 2023Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Cluttercited by 22
  23. 2021Discriminative Triad Matching and Reconstruction for Weakly Referring Expression Groundingcited by 21
  24. 2022Adapting CLIP For Phrase Localization Without Further Trainingcited by 21
  25. 2022Skimming, Locating, then Perusing: A Human-Like Framework for Natural Language Video Localizationcited by 21
  26. 2021Iterative Shrinking for Referring Expression Grounding Using Deep Reinforcement Learningcited by 20
  27. 2022Ground then Navigate: Language-guided Navigation in Dynamic Scenescited by 20
  28. 2022End-to-End Modeling via Information Tree for One-Shot Natural Language Spatial Video Groundingcited by 19
  29. 2023A Survey on Video Moment Localizationcited by 18
  30. 2023Distilling Coarse-to-Fine Semantic Matching Knowledge for Weakly Supervised 3D Visual Groundingcited by 18
  31. 2021Disentangled Motif-aware Graph Learning for Phrase Groundingcited by 17
  32. 2022YORO -- Lightweight End to End Visual Groundingcited by 17
  33. 2021Who's Waldo? Linking People Across Text and Imagescited by 16
  34. 2022Toward Explainable and Fine-Grained 3D Grounding through Referring Textual Phrasescited by 16
  35. 2022Understanding Embodied Reference with Touch-Line Transformercited by 16
  36. 2022MNER-QG: An End-to-End MRC framework for Multimodal Named Entity Recognition with Query Groundingcited by 16
  37. 2022Fully and Weakly Supervised Referring Expression Segmentation with End-to-End Learningcited by 14
  38. 2023TextPSG: Panoptic Scene Graph Generation from Textual Descriptionscited by 14

50 most cited of 112 All 112 (JSON)