# Multimodal Referring Segmentation: A Survey

arXiv: https://arxiv.org/abs/2508.00265 · HTML: https://mlsanity.com/paper/2508.00265/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 317 of its 405 references that are in the map.
- First posted on arXiv 1 Aug 2025.

## Builds on (most cited first, 30 of 317)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)

## Built on by (most cited first, 22 of 22)

- [MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2508.05630/) (2025)
- [MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation](https://mlsanity.com/paper/2512.10945/) (2025)
- [ReferSplat: Referring Segmentation in 3D Gaussian Splatting](https://mlsanity.com/paper/2508.08252/) (2025)
- [A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generation](https://mlsanity.com/paper/2508.09977/) (2025)
- [GREx: Generalized Referring Expression Segmentation, Comprehension, and Generation](https://mlsanity.com/paper/2601.05244/) (2026)
- [LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generation](https://mlsanity.com/paper/2604.11789/) (2026)
- [SegCompass: Exploring Interpretable Alignment with Sparse Autoencoders for Enhanced Reasoning Segmentation](https://mlsanity.com/paper/2605.22658/) (2026)
- [Watch, Remember, Reason: Human-View Video Understanding with MLLMs](https://mlsanity.com/paper/2606.07433/) (2026)
- [ROSE: Retrieval-Oriented Segmentation Enhancement](https://mlsanity.com/paper/2604.14147/) (2026)
- [Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Model](https://mlsanity.com/paper/2509.07825/) (2025)
- [Audit After Segmentation: Reference-Free Mask Quality Assessment for Language-Referred Audio-Visual Segmentation](https://mlsanity.com/paper/2602.03892/) (2026)
- [Multi-label Instance-level Generalised Visual Grounding in Agriculture](https://mlsanity.com/paper/2603.06699/) (2026)
- [Ref-SAM3D: Bridging SAM3D with Text for Reference 3D Reconstruction](https://mlsanity.com/paper/2511.19426/) (2025)
- [3D Segmentation Using Viewpoint-Dependent Spatial Relationships](https://mlsanity.com/paper/2605.15708/) (2026)
- [B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation](https://mlsanity.com/paper/2605.23500/) (2026)
- [Progressive Prompt-Guided Cross-Modal Reasoning for Referring Image Segmentation](https://mlsanity.com/paper/2603.27993/) (2026)
- [A Paradigm Shift: Fully End-to-End Training for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2604.02860/) (2026)
- [Seek to Segment: Active Perception for Panoramic Referring Segmentation](https://mlsanity.com/paper/2607.02497/) (2026)
- [SAM-MT: Real-Time Interactive Multi-Target Video Segmentation](https://mlsanity.com/paper/2607.08688/) (2026)
- [Motion-Aware Reasoning from Speech to Mask Tracks: Runner-up Solution for the MeViS-Audio Track of the 8th LSVOS Challenge 2026](https://mlsanity.com/paper/2608.22337/) (2026)
- [Position Is All You Need: A Free Lunch Token Compression Strategy for MLLM-based Referring Expression Segmentation](https://mlsanity.com/paper/2608.26142/) (2026)
- [DeCo: Efficient Decouple-to-Couple Learning for Multi-Task Visual Grounding](https://mlsanity.com/paper/2609.24409/) (2026)

## Abstract (arXiv)

Multimodal referring segmentation aims to segment target objects in visual scenes, such as images, videos, and 3D scenes, based on referring expressions in text or audio format. This task plays a crucial role in practical applications requiring accurate object perception based on user instructions. Over the past decade, it has gained significant attention in the multimodal community, driven by advances in convolutional neural networks, transformers, and large language models, all of which have substantially improved multimodal perception capabilities. This paper provides a comprehensive survey of multimodal referring segmentation. We begin by introducing this field's background, including problem definitions and commonly used datasets. Next, we summarize a unified meta architecture for referring segmentation and review representative methods across three primary visual scenes, including images, videos, and 3D scenes. We further discuss Generalized Referring Expression (GREx) methods to address the challenges of real-world complexity, along with related tasks and practical applications. Extensive performance comparisons on standard benchmarks are also provided. We continually track related works at https://github.com/henghuiding/Awesome-Multimodal-Referring-Segmentation.