# Towards Complex-query Referring Image Segmentation: A Novel Benchmark

arXiv: https://arxiv.org/abs/2309.17205 · HTML: https://mlsanity.com/paper/2309.17205/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 65 references that are in the map.
- First posted on arXiv 29 Sep 2023.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [Unleashing Text-to-Image Diffusion Models for Visual Perception](https://mlsanity.com/paper/2303.02153/) (2023)
- [Cross-Modal Self-Attention Network for Referring Image Segmentation](https://mlsanity.com/paper/1904.04745/) (2019)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2108.05565/) (2021)
- [Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2003.08813/) (2020)
- [A Fast and Accurate One-Stage Approach to Visual Grounding](https://mlsanity.com/paper/1908.06354/) (2019)
- [Universal Instance Perception as Object Discovery and Retrieval](https://mlsanity.com/paper/2303.06674/) (2023)
- [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](https://mlsanity.com/paper/2003.12962/) (2020)

## Built on by (most cited first, 10 of 10)

- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenes](https://mlsanity.com/paper/2502.00392/) (2025)
- [FORTIS: Benchmarking Over-Privilege in Agent Skills](https://mlsanity.com/paper/2605.09163/) (2026)
- [Charts Are Not Images: On the Challenges of Scientific Chart Editing](https://mlsanity.com/paper/2512.00752/) (2025)
- [Context-Enhanced Video Moment Retrieval with Large Language Models](https://mlsanity.com/paper/2405.12540/) (2024)
- [Domain-wise Invariant Learning for Panoptic Scene Graph Generation](https://mlsanity.com/paper/2310.05867/) (2023)
- [Backpropagation-Free Multi-modal On-Device Model Adaptation via Cloud-Device Collaboration](https://mlsanity.com/paper/2406.01601/) (2024)
- [Grounding is All You Need? Dual Temporal Grounding for Video Dialog](https://mlsanity.com/paper/2410.05767/) (2024)
- [Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentation](https://mlsanity.com/paper/2509.13676/) (2025)
- [Planner-Refiner: Dynamic Space-Time Refinement for Vision-Language Alignment in Videos](https://mlsanity.com/paper/2508.07330/) (2025)

## Abstract (arXiv)

Referring Image Understanding (RIS) has been extensively studied over the past decade, leading to the development of advanced algorithms. However, there has been a lack of research investigating how existing algorithms should be benchmarked with complex language queries, which include more informative descriptions of surrounding objects and backgrounds (eg textit"the black car." vs. textit"the black car is parking on the road and beside the bus."). Given the significant improvement in the semantic understanding capability of large pre-trained models, it is crucial to take a step further in RIS by incorporating complex language that resembles real-world applications. To close this gap, building upon the existing RefCOCO and Visual Genome datasets, we propose a new RIS benchmark with complex queries, namely RIS-CQ. The RIS-CQ dataset is of high quality and large scale, which challenges the existing RIS with enriched, specific and informative queries, and enables a more realistic scenario of RIS research. Besides, we present a nichetargeting method to better task the RIS-CQ, called dual-modality graph alignment model (textbftextscDuMoGa), which outperforms a series of RIS methods.