mlsanityData to 14 Oct 2026

A Joint Speaker-Listener-Reinforcer Model for Referring Expressions

arXiv 1612.09542 ↗First posted 30 Dec 2016cs.CV cs.AI cs.CLcited by 74

Not on the map. Why?

Abstract (arXiv)

Referring expressions are natural language constructions used to identify particular objects within a scene. In this paper, we propose a unified framework for the tasks of referring expression comprehension and generation. Our model is composed of three modules: speaker, listener, and reinforcer. The speaker generates referring expressions, the listener comprehends referring expressions, and the reinforcer introduces a reward function to guide sampling of more discriminative expressions. The listener-speaker modules are trained jointly in an end-to-end learning framework, allowing the modules to be aware of one another during learning while also benefiting from the discriminative reinforcer's feedback. We demonstrate that this unified framework and training achieves state-of-the-art results for both comprehension and generation on three referring expression datasets. Project and demo page: https://vision.cs.unc.edu/refer

Key facts. Cited by 74 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 18; 2022: 19; 2023: 17; 2024: 11; 2025: 7; 2026: 2.

74 papers.

Sort:

By area:

  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  2. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  3. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  4. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  5. 2021Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentationcited by 111
  6. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 98
  7. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 98
  8. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  9. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  10. 2021Look Before You Leap: Learning Landmark Features for One-Stage Visual Groundingcited by 52
  11. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  12. 2024Toward Visual Grounding: A Surveycited by 52
38 more
  1. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 49
  2. 2022Instance-Specific Feature Propagation for Referring Segmentationcited by 45
  3. 2021Visual Grounding with Transformerscited by 42
  4. 2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
  5. 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
  6. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39
  7. 2023Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentationcited by 36
  8. 20233D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentationcited by 36
  9. 2021Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Imagescited by 33
  10. 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referringcited by 29
  11. 2023Parallel Vertex Diffusion for Unified Visual Groundingcited by 28
  12. 2022Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Groundingcited by 22
  13. 2022Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Taskscited by 21
  14. 2021Composing Pick-and-Place Tasks By Grounding Languagecited by 20
  15. 2021Iterative Shrinking for Referring Expression Grounding Using Deep Reinforcement Learningcited by 20
  16. 2023HandMeThat: Human-Robot Communication in Physical and Social Environmentscited by 20
  17. 2022Phrase-Based Affordance Detection via Cyclic Bilateral Interactioncited by 19
  18. 20243D-GRES: Generalized 3D Referring Expression Segmentationcited by 18
  19. 2025Towards Visual Text Grounding of Multimodal Large Language Modelcited by 18
  20. 2022YORO -- Lightweight End to End Visual Groundingcited by 17
  21. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  22. 2021Reference-Centric Models for Grounded Collaborative Dialoguecited by 15
  23. 2021Connecting What to Say With Where to Look by Modeling Human Attention Tracescited by 13
  24. 2023Towards Real-Time Panoptic Narrative Grounding by an End-to-End Grounding Networkcited by 13
  25. 2024ControlCap: Controllable Region-level Captioningcited by 13
  26. 2021Co-Grounding Networks with Semantic Attention for Referring Expression Comprehension in Videoscited by 11
  27. 2022Correspondence Matters for Video Referring Expression Comprehensioncited by 10
  28. 2021Unified Questioner Transformer for Descriptive Question Generation in Goal-Oriented Visual Dialoguecited by 8
  29. 2023DisCLIP: Open-Vocabulary Referring Expression Generationcited by 8
  30. 2022Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matchingcited by 7
  31. 2022Guiding Visual Question Answering with Attention Priorscited by 7
  32. 2023Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mindcited by 7
  33. 2021Giving Commands to a Self-Driving Car: How to Deal with Uncertain Situations?cited by 6
  34. 2022Towards Unifying Reference Expression Generation and Comprehensioncited by 6
  35. 2023Cycle-Consistency Learning for Captioning and Groundingcited by 6
  36. 2025Referring Expression Comprehension for Small Objectscited by 6
  37. 2026GREx: Generalized Referring Expression Segmentation, Comprehension, and Generationcited by 6
  38. 2021Proposal-free One-stage Referring Expression via Grid-Word Cross-Attentioncited by 5

50 most cited of 74 All 74 (JSON)