# Relationship-Embedded Representation Learning for Grounding Referring Expressions

arXiv: https://arxiv.org/abs/1906.04464 · HTML: https://mlsanity.com/paper/1906.04464/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2019.

## Built on by (most cited first, 25 of 25)

- [RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data](https://mlsanity.com/paper/2210.12634/) (2022)
- [Cross-Modal Progressive Comprehension for Referring Segmentation](https://mlsanity.com/paper/2105.07175/) (2021)
- [Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloud](https://mlsanity.com/paper/2103.16381/) (2021)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [Contrastive Grouping with Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2309.01017/) (2023)
- [Scene-Intuitive Agent for Remote Embodied Visual Grounding](https://mlsanity.com/paper/2103.12944/) (2021)
- [Visual Grounding with Transformers](https://mlsanity.com/paper/2105.04281/) (2021)
- [TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformer](https://mlsanity.com/paper/2206.06619/) (2022)
- [CoTDet: Affordance Knowledge Prompting for Task Driven Object Detection](https://mlsanity.com/paper/2309.01093/) (2023)
- [Temporal Collection and Distribution for Referring Video Object Segmentation](https://mlsanity.com/paper/2309.03473/) (2023)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation](https://mlsanity.com/paper/2404.11998/) (2024)
- [Mono3DVG: 3D Visual Grounding in Monocular Images](https://mlsanity.com/paper/2312.08022/) (2023)
- [Bootstrapping Referring Multi-Object Tracking](https://mlsanity.com/paper/2406.05039/) (2024)
- [Zero-shot Generalizable Incremental Learning for Vision-Language Object Detection](https://mlsanity.com/paper/2403.01680/) (2024)
- [Learning to Evaluate Performance of Multi-modal Semantic Localization](https://mlsanity.com/paper/2209.06515/) (2022)
- [Spatial and Visual Perspective-Taking via View Rotation and Relation Reasoning for Embodied Reference Understanding](https://mlsanity.com/paper/2309.01073/) (2023)
- [Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context](https://mlsanity.com/paper/2510.20229/) (2025)
- [Incremental Object Grounding Using Scene Graphs](https://mlsanity.com/paper/2201.01901/) (2022)
- [Exploring Sparse Spatial Relation in Graph Inference for Text-Based VQA](https://mlsanity.com/paper/2310.09147/) (2023)
- [Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats](https://mlsanity.com/paper/2511.17254/) (2025)
- [Hierarchical Aligned Multimodal Learning for NER on Tweet Posts](https://mlsanity.com/paper/2305.08372/) (2023)
- [Phrase Decoupling Cross-Modal Hierarchical Matching and Progressive Position Correction for Visual Grounding](https://mlsanity.com/paper/2410.23570/) (2024)
- [B2N3D: Progressive Learning from Binary to N-ary Relationships for 3D Object Grounding](https://mlsanity.com/paper/2510.10194/) (2025)
- [Attribute Retrieving for Open-Vocabulary Endoscopic Compositional Referring Segmentation](https://mlsanity.com/paper/2607.08397/) (2026)

## Abstract (arXiv)

Grounding referring expressions in images aims to locate the object instance in an image described by a referring expression. It involves a joint understanding of natural language and image content, and is essential for a range of visual tasks related to human-computer interaction. As a language-to-vision matching task, the core of this problem is to not only extract all the necessary information (i.e., objects and the relationships among them) in both the image and referring expression, but also make full use of context information to align cross-modal semantic concepts in the extracted information. Unfortunately, existing work on grounding referring expressions fails to accurately extract multi-order relationships from the referring expression and associate them with the objects and their related contexts in the image. In this paper, we propose a Cross-Modal Relationship Extractor (CMRE) to adaptively highlight objects and relationships (spatial and semantic relations) related to the given expression with a cross-modal attention mechanism, and represent the extracted information as a language-guided visual relation graph. In addition, we propose a Gated Graph Convolutional Network (GGCN) to compute multimodal semantic contexts by fusing information from different modes and propagating multimodal information in the structured relation graph. Experimental results on three common benchmark datasets show that our Cross-Modal Relationship Inference Network, which consists of CMRE and GGCN, significantly surpasses all existing state-of-the-art methods. Code is available at https://github.com/sibeiyang/sgmn/tree/master/lib/cmrin_models