# Dynamic Graph Attention for Referring Expression Comprehension

arXiv: https://arxiv.org/abs/1909.08164 · HTML: https://mlsanity.com/paper/1909.08164/

## Key facts

- Cited by 111 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2019.

## Built on by (most cited first, 30 of 111)

- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [What does CLIP know about a red circle? Visual prompt engineering for VLMs](https://mlsanity.com/paper/2304.06712/) (2023)
- [Universal Instance Perception as Object Discovery and Retrieval](https://mlsanity.com/paper/2303.06674/) (2023)
- [SeqTR: A Simple yet Universal Network for Visual Grounding](https://mlsanity.com/paper/2203.16265/) (2022)
- [RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data](https://mlsanity.com/paper/2210.12634/) (2022)
- [3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection](https://mlsanity.com/paper/2204.06272/) (2022)
- [Cross-Modal Progressive Comprehension for Referring Segmentation](https://mlsanity.com/paper/2105.07175/) (2021)
- [TubeDETR: Spatio-Temporal Video Grounding with Transformers](https://mlsanity.com/paper/2203.16434/) (2022)
- [LanguageRefer: Spatial-Language Model for 3D Visual Grounding](https://mlsanity.com/paper/2107.03438/) (2021)
- [Open-vocabulary Object Segmentation with Diffusion Models](https://mlsanity.com/paper/2301.05221/) (2023)
- [Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloud](https://mlsanity.com/paper/2103.16381/) (2021)
- [Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoning](https://mlsanity.com/paper/2205.00272/) (2022)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [Contrastive Grouping with Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2309.01017/) (2023)
- [Shifting More Attention to Visual Backbone: Query-modulated Refinement Networks for End-to-End Visual Grounding](https://mlsanity.com/paper/2203.15442/) (2022)
- [Look Before You Leap: Learning Landmark Features for One-Stage Visual Grounding](https://mlsanity.com/paper/2104.04386/) (2021)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Instance-Specific Feature Propagation for Referring Segmentation](https://mlsanity.com/paper/2204.12109/) (2022)
- [TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformer](https://mlsanity.com/paper/2206.06619/) (2022)
- [Temporal Sentence Grounding in Videos: A Survey and Future Directions](https://mlsanity.com/paper/2201.08071/) (2022)
- [CoTDet: Affordance Knowledge Prompting for Task Driven Object Detection](https://mlsanity.com/paper/2309.01093/) (2023)
- [SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion](https://mlsanity.com/paper/2409.17531/) (2024)
- [Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Images](https://mlsanity.com/paper/2103.07894/) (2021)
- [HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding](https://mlsanity.com/paper/2404.13400/) (2024)
- [Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Grounding](https://mlsanity.com/paper/2209.13959/) (2022)
- [Parallel Vertex Diffusion for Unified Visual Grounding](https://mlsanity.com/paper/2303.07216/) (2023)
- [Temporal Collection and Distribution for Referring Video Object Segmentation](https://mlsanity.com/paper/2309.03473/) (2023)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Context Disentangling and Prototype Inheriting for Robust Visual Grounding](https://mlsanity.com/paper/2312.11967/) (2023)
- [Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation](https://mlsanity.com/paper/2404.11998/) (2024)

## Abstract (arXiv)

Referring expression comprehension aims to locate the object instance described by a natural language referring expression in an image. This task is compositional and inherently requires visual reasoning on top of the relationships among the objects in the image. Meanwhile, the visual reasoning process is guided by the linguistic structure of the referring expression. However, existing approaches treat the objects in isolation or only explore the first-order relationships between objects without being aligned with the potential complexity of the expression. Thus it is hard for them to adapt to the grounding of complex referring expressions. In this paper, we explore the problem of referring expression comprehension from the perspective of language-driven visual reasoning, and propose a dynamic graph attention network to perform multi-step reasoning by modeling both the relationships among the objects in the image and the linguistic structure of the expression. In particular, we construct a graph for the image with the nodes and edges corresponding to the objects and their relationships respectively, propose a differential analyzer to predict a language-guided visual reasoning process, and perform stepwise reasoning on top of the graph to update the compound object representation at every node. Experimental results demonstrate that the proposed method can not only significantly surpass all existing state-of-the-art algorithms across three common benchmark datasets, but also generate interpretable visual evidences for stepwisely locating the objects referred to in complex language descriptions.