mlsanityData to 14 Oct 2026

GRES: Generalized Referring Expression Segmentation

arXiv 2306.00968 ↗First posted 1 Jun 2023cs.CVcited by 254

Not on the map. Why?

Cited by 70 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Referring Expression Segmentation (RES) aims to generate a segmentation mask for the object described by a given language expression. Existing classic RES datasets and methods commonly support single-target expressions only, i.e., one expression refers to one target object. Multi-target and no-target expressions are not considered. This limits the usage of RES in practice. In this paper, we introduce a new benchmark called Generalized Referring Expression Segmentation (GRES), which extends the classic RES to allow expressions to refer to an arbitrary number of target objects. Towards this, we construct the first large-scale GRES dataset called gRefCOCO that contains multi-target, no-target, and single-target expressions. GRES and gRefCOCO are designed to be well-compatible with RES, facilitating extensive experiments to study the performance gap of the existing RES methods on the GRES task. In the experimental study, we find that one of the big challenges of GRES is complex relationship modeling. Based on this, we propose a region-based GRES baseline ReLA that adaptively divides the image into regions with sub-instance clues, and explicitly models the region-region and region-language dependencies. The proposed approach ReLA achieves new state-of-the-art performance on the both newly proposed GRES and classic RES tasks. The proposed gRefCOCO dataset and method are available at https://henghuiding.github.io/GRES.

Key facts. Cited by 254 papers in the map (data through 14 Oct 2026). Builds on 50 of its 67 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

50 of 67 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  11. 2016Modeling Context in Referring Expressionscited by 1,063
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
38 more
  1. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  2. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  3. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  4. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  5. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  6. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  7. 2016Segmentation from Natural Language Expressionscited by 257
  8. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  9. 2023MOSE: A New Dataset for Video Object Segmentation in Complex Scenescited by 234
  10. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  11. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 195
  12. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 140
  13. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 138
  14. 2020Referring Image Segmentation via Cross-Modal Progressive Comprehensioncited by 131
  15. 2015Natural Language Object Retrievalcited by 124
  16. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  17. 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116
  18. 2022ReSTR: Convolution-free Referring Image Segmentation Using Transformerscited by 112
  19. 2019A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehensioncited by 111
  20. 2021Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentationcited by 111
  21. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 111
  22. 2016Modeling Relationships in Referential Expressions with Compositional Modular Networkscited by 106
  23. 2018Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networkscited by 102
  24. 2020Linguistic Structure Guided Context Modeling for Referring Image Segmentationcited by 102
  25. 2021Cross-Modal Progressive Comprehension for Referring Segmentationcited by 102
  26. 2021Locate then Segment: A Strong Pipeline for Referring Image Segmentationcited by 98
  27. 2021ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentationcited by 93
  28. 2018Dynamic Multimodal Instance Segmentation guided by natural language queriescited by 83
  29. 2016A Joint Speaker-Listener-Reinforcer Model for Referring Expressionscited by 74
  30. 2017Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queriescited by 56
  31. 2017Comprehension-guided referring expressionscited by 52
  32. 2019Referring Expression Object Segmentation with Caption-Aware Consistencycited by 46
  33. 2022Instance-Specific Feature Propagation for Referring Segmentationcited by 45
  34. 2023Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentationcited by 36
  35. 2022CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentationcited by 28
  36. 2023Prototype Adaption and Projection for Few- and Zero-shot 3D Point Cloud Semantic Segmentationcited by 26
  37. 2017Discriminative Bimodal Networks for Visual Localization and Detection with Natural Language Queriescited by 17
  38. 2023Semantic-Promoted Debiasing and Background Disambiguation for Zero-Shot Instance Segmentationcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 32; 2024: 59; 2025: 94; 2026: 69.

254 papers.

Sort:

By area:

  1. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  2. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  3. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  4. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 239
  5. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 228
  6. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 222
  7. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 177
  8. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  9. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 164
  10. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  11. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 154
  12. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
38 more
  1. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 127
  2. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  3. 2023Language Prompt for Autonomous Drivingcited by 111
  4. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 101
  5. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 100
  6. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
  7. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 80
  8. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 71
  9. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  10. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  11. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 69
  12. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 65
  13. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 63
  14. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 61
  15. 2025LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Modelscited by 57
  16. 2024Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentationcited by 55
  17. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  18. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 53
  19. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  20. 2024LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoningcited by 52
  21. 2024Toward Visual Grounding: A Surveycited by 52
  22. 2025VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learningcited by 51
  23. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 51
  24. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  25. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  26. 2025Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learningcited by 44
  27. 2023GREC: Generalized Referring Expression Comprehensioncited by 42
  28. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 41
  29. 2023Mask Grounding for Referring Image Segmentationcited by 40
  30. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39
  31. 2025RemoteSAM: Towards Segment Anything for Earth Observationcited by 39
  32. 2024ReMamber: Referring Image Segmentation with Mamba Twistercited by 38
  33. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  34. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 35
  35. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  36. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 35
  37. 2023Ref-Diff: Zero-shot Referring Image Segmentation with Generative Modelscited by 33
  38. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33

50 most cited of 254 All 254 (JSON)