# Referring Expression Comprehension: A Survey of Methods and Datasets

arXiv: https://arxiv.org/abs/2007.09554 · HTML: https://mlsanity.com/paper/2007.09554/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jul 2020.

## Built on by (most cited first, 30 of 79)

- [Alpha-CLIP: A CLIP Model Focusing on Wherever You Want](https://mlsanity.com/paper/2312.03818/) (2023)
- [VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents](https://mlsanity.com/paper/2408.06327/) (2024)
- [VL-Mamba: Exploring State Space Models for Multimodal Learning](https://mlsanity.com/paper/2403.13600/) (2024)
- [UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning](https://mlsanity.com/paper/2505.14231/) (2025)
- [Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloud](https://mlsanity.com/paper/2103.16381/) (2021)
- [MyVLM: Personalizing VLMs for User-Specific Queries](https://mlsanity.com/paper/2403.14599/) (2024)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Visual Grounding with Transformers](https://mlsanity.com/paper/2105.04281/) (2021)
- [CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding](https://mlsanity.com/paper/2305.08685/) (2023)
- [LocCa: Visual Pretraining with Location-aware Captioners](https://mlsanity.com/paper/2403.19596/) (2024)
- [INVIGORATE: Interactive Visual Grounding and Grasping in Clutter](https://mlsanity.com/paper/2108.11092/) (2021)
- [HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding](https://mlsanity.com/paper/2404.13400/) (2024)
- [OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling](https://mlsanity.com/paper/2410.08021/) (2024)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chains](https://mlsanity.com/paper/2405.10620/) (2024)
- [A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directions](https://mlsanity.com/paper/2406.05785/) (2024)
- [Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter](https://mlsanity.com/paper/2311.05779/) (2023)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [Referring to Any Person](https://mlsanity.com/paper/2503.08507/) (2025)
- [Who's Waldo? Linking People Across Text and Images](https://mlsanity.com/paper/2108.07253/) (2021)
- [Dense Object Grounding in 3D Scenes](https://mlsanity.com/paper/2309.02224/) (2023)
- [FindIt: Generalized Localization with Natural Language Queries](https://mlsanity.com/paper/2203.17273/) (2022)
- [WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radar](https://mlsanity.com/paper/2403.12686/) (2024)
- [Answer-Me: Multi-Task Open-Vocabulary Visual Question Answering](https://mlsanity.com/paper/2205.00949/) (2022)
- [ScanERU: Interactive 3D Visual Grounding based on Embodied Reference Understanding](https://mlsanity.com/paper/2303.13186/) (2023)
- [CoLLIE: Continual Learning of Language Grounding from Language-Image Embeddings](https://mlsanity.com/paper/2111.07993/) (2021)
- [Toloka Visual Question Answering Benchmark](https://mlsanity.com/paper/2309.16511/) (2023)
- [Grounding-Aware Token Pruning: Recovering from Drastic Performance Drops in Visual Grounding Caused by Pruning](https://mlsanity.com/paper/2506.21873/) (2025)
- [GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation](https://mlsanity.com/paper/2512.17495/) (2025)
- [Evaluation of Audio-Visual Alignments in Visually Grounded Speech Models](https://mlsanity.com/paper/2108.02562/) (2021)

## Abstract (arXiv)

Referring expression comprehension (REC) aims to localize a target object in an image described by a referring expression phrased in natural language. Different from the object detection task that queried object labels have been pre-defined, the REC problem only can observe the queries during the test. It thus more challenging than a conventional computer vision problem. This task has attracted a lot of attention from both computer vision and natural language processing community, and several lines of work have been proposed, from CNN-RNN model, modular network to complex graph-based model. In this survey, we first examine the state of the art by comparing modern approaches to the problem. We classify methods by their mechanism to encode the visual and textual modalities. In particular, we examine the common approach of joint embedding images and expressions to a common feature space. We also discuss modular architectures and graph-based models that interface with structured graph representation. In the second part of this survey, we review the datasets available for training and evaluating REC systems. We then group results according to the datasets, backbone models, settings so that they can be fairly compared. Finally, we discuss promising future directions for the field, in particular the compositional referring expression comprehension that requires longer reasoning chain to address.