mlsanityData to 7 Oct 2026

Comprehension-guided referring expressions

arXiv 1701.03439 ↗First posted 12 Jan 2017cs.CVcited by 52

Not on the map. Why?

Abstract (arXiv)

We consider generation and comprehension of natural language referring expression for objects in an image. Unlike generic "image captioning" which lacks natural standard evaluation criteria, quality of a referring expression may be measured by the receiver's ability to correctly infer which object is being described. Following this intuition, we propose two approaches to utilize models trained for comprehension task to generate better expressions. First, we use a comprehension module trained on human-generated expressions, as a "critic" of referring expression generator. The comprehension module serves as a differentiable proxy of human evaluation, providing training signal to the generation module. Second, we use the comprehension module in a generate-and-rerank pipeline, which chooses from candidate expressions generated by a model according to their performance on the comprehension task. We show that both approaches lead to improved referring expression generation on multiple benchmark datasets.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jan 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 17; 2023: 14; 2024: 4; 2025: 3; 2026: 2.

52 papers.

Sort:

By area:

  1. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  2. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  3. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  4. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  5. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  6. 2021Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentationcited by 111
  7. 2023Referring Multi-Object Trackingcited by 91
  8. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  9. 2024Toward Visual Grounding: A Surveycited by 52
  10. 2022Instance-Specific Feature Propagation for Referring Segmentationcited by 45
  11. 2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
  12. 2023Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentationcited by 36
38 more
  1. 2023Parallel Vertex Diffusion for Unified Visual Groundingcited by 28
  2. 2023Context Disentangling and Prototype Inheriting for Robust Visual Groundingcited by 23
  3. 2022Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Groundingcited by 22
  4. 2023Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Cluttercited by 22
  5. 2022End-to-End Modeling via Information Tree for One-Shot Natural Language Spatial Video Groundingcited by 19
  6. 2025Towards Visual Text Grounding of Multimodal Large Language Modelcited by 18
  7. 2021Disentangled Motif-aware Graph Learning for Phrase Groundingcited by 17
  8. 2022YORO -- Lightweight End to End Visual Groundingcited by 16
  9. 2023Towards Real-Time Panoptic Narrative Grounding by an End-to-End Grounding Networkcited by 13
  10. 2024Grounding-IQA: Grounding Multimodal Language Model for Image Quality Assessmentcited by 12
  11. 2021Unified Questioner Transformer for Descriptive Question Generation in Goal-Oriented Visual Dialoguecited by 8
  12. 2021Evaluation of Audio-Visual Alignments in Visually Grounded Speech Modelscited by 8
  13. 2022Emergence of hierarchical reference systems in multi-agent communicationcited by 8
  14. 2023DisCLIP: Open-Vocabulary Referring Expression Generationcited by 8
  15. 2023NICE: Improving Panoptic Narrative Detection and Segmentation with Cascading Collaborative Learningcited by 8
  16. 2021Target-dependent UNITER: A Transformer-Based Multimodal Language Comprehension Model for Domestic Service Robotscited by 7
  17. 2022Towards Unifying Reference Expression Generation and Comprehensioncited by 6
  18. 2025Referring Expression Comprehension for Small Objectscited by 6
  19. 2022A Survivor in the Era of Large-Scale Pretraining: An Empirical Study of One-Stage Referring Expression Comprehensioncited by 5
  20. 2021Understanding Synonymous Referring Expressions via Contrastive Featurescited by 4
  21. 2022Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attentioncited by 4
  22. 2022Find Someone Who: Visual Commonsense Understanding in Human-Centric Groundingcited by 4
  23. 2023Pragmatic Inference with a CLIP Listener for Contrastive Captioningcited by 4
  24. 2021Decoupled Spatial Temporal Graphs for Generic Visual Groundingcited by 3
  25. 2025AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarioscited by 3
  26. 2022Sim-To-Real Transfer of Visual Grounding for Human-Aided Ambiguity Resolutioncited by 2
  27. 2022RefCrowd: Grounding the Target in Crowd with Referring Expressionscited by 2
  28. 2024Resilience through Scene Context in Visual Referring Expression Generationcited by 2
  29. 2021Image interpretation by iterative bottom-up top-down processingcited by 1
  30. 2021Goal-driven text descriptions for imagescited by 1
  31. 2023Pento-DIARef: A Diagnostic Dataset for Learning the Incremental Algorithm for Referring Expression Generation from Examplescited by 1
  32. 2023Continual Referring Expression Comprehension via Dual Modular Memorizationcited by 1
  33. 2024Referring Expression Generation in Visually Grounded Dialogue with Discourse-aware Comprehension Guidingcited by 1
  34. 2026Chain-of-Caption: Training-free improvement of multimodal large language model on referring expression comprehensioncited by 1
  35. 2022Differentiated Relevances Embedding for Group-based Referring Expression Comprehensioncited by 0
  36. 2022Referring Expressions with Rational Speech Act Framework: A Probabilistic Approachcited by 0
  37. 2022The Role of Explanatory Value in Natural Language Processingcited by 0
  38. 2022Enhancing Interpretability and Interactivity in Robot Manipulation: A Neurosymbolic Approachcited by 0

50 most cited of 52 All 52 (JSON)