# 3D-GRES: Generalized 3D Referring Expression Segmentation

arXiv: https://arxiv.org/abs/2407.20664 · HTML: https://mlsanity.com/paper/2407.20664/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 79 references that are in the map.
- First posted on arXiv 30 Jul 2024.

## Builds on (most cited first, 30 of 57)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs](https://mlsanity.com/paper/1711.09869/) (2017)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [RemoteCLIP: A Vision Language Foundation Model for Remote Sensing](https://mlsanity.com/paper/2306.11029/) (2023)
- [X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval](https://mlsanity.com/paper/2207.07285/) (2022)
- [Cross-Modal Self-Attention Network for Referring Image Segmentation](https://mlsanity.com/paper/1904.04745/) (2019)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2108.05565/) (2021)
- [Group-Free 3D Object Detection via Transformers](https://mlsanity.com/paper/2104.00678/) (2021)
- [Multi3DRefer: Grounding Text Description to Multiple 3D Objects](https://mlsanity.com/paper/2309.05251/) (2023)
- [InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring](https://mlsanity.com/paper/2103.01128/) (2021)
- [Learning to Assemble Neural Module Tree Networks for Visual Grounding](https://mlsanity.com/paper/1812.03299/) (2018)
- [Improving One-stage Visual Grounding by Recursive Sub-query Construction](https://mlsanity.com/paper/2008.01059/) (2020)
- [Natural Language Object Retrieval](https://mlsanity.com/paper/1511.04164/) (2015)
- [Language Conditioned Spatial Relation Reasoning for 3D Object Grounding](https://mlsanity.com/paper/2211.09646/) (2022)
- [Superpoint Transformer for 3D Scene Instance Segmentation](https://mlsanity.com/paper/2211.15766/) (2022)
- [VLT: Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2210.15871/) (2022)
- [3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection](https://mlsanity.com/paper/2204.06272/) (2022)
- [Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networks](https://mlsanity.com/paper/1812.04794/) (2018)

## Built on by (most cited first, 18 of 18)

- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension](https://mlsanity.com/paper/2501.01416/) (2025)
- [RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation](https://mlsanity.com/paper/2412.02402/) (2024)
- [Multimodal 3D Reasoning Segmentation with Complex Scenes](https://mlsanity.com/paper/2411.13927/) (2024)
- [MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentation](https://mlsanity.com/paper/2601.06874/) (2026)
- [LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generation](https://mlsanity.com/paper/2604.11789/) (2026)
- [PVUW 2025 Challenge Report: Advances in Pixel-level Understanding of Complex Videos in the Wild](https://mlsanity.com/paper/2504.11326/) (2025)
- [LSVOS 2025 Challenge Report: Recent Advances in Complex Video Object Segmentation](https://mlsanity.com/paper/2510.11063/) (2025)
- [3DResT: A Strong Baseline for Semi-Supervised 3D Referring Expression Segmentation](https://mlsanity.com/paper/2504.12599/) (2025)
- [Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Model](https://mlsanity.com/paper/2509.07825/) (2025)
- [LIHE: Linguistic Instance-Split Hyperbolic-Euclidean Framework for Generalized Weakly-Supervised Referring Expression Comprehension](https://mlsanity.com/paper/2511.12020/) (2025)
- [3D-DRES: Detailed 3D Referring Expression Segmentation](https://mlsanity.com/paper/2603.02896/) (2026)
- [3D Segmentation Using Viewpoint-Dependent Spatial Relationships](https://mlsanity.com/paper/2605.15708/) (2026)
- [TrackRef3D: Multi-View Consistent Track-then-Label for Open-World Referring Segmentation in 3D Gaussian Splatting](https://mlsanity.com/paper/2605.26576/) (2026)
- [Proxy-Embedding as an Adversarial Teacher: An Embedding-Guided Bidirectional Attack for Referring Expression Segmentation Models](https://mlsanity.com/paper/2506.16157/) (2025)
- [DenseScan: Advancing 3D Scene Understanding with 2D Dense Annotation](https://mlsanity.com/paper/2512.00226/) (2025)
- [Hierarchical Collaborative Fusion for 3D Instance-aware Referring Expression Segmentation](https://mlsanity.com/paper/2603.06250/) (2026)
- [Grounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances](https://mlsanity.com/paper/2605.11616/) (2026)

## Abstract (arXiv)

3D Referring Expression Segmentation (3D-RES) is dedicated to segmenting a specific instance within a 3D space based on a natural language description. However, current approaches are limited to segmenting a single target, restricting the versatility of the task. To overcome this limitation, we introduce Generalized 3D Referring Expression Segmentation (3D-GRES), which extends the capability to segment any number of instances based on natural language instructions. In addressing this broader task, we propose the Multi-Query Decoupled Interaction Network (MDIN), designed to break down multi-object segmentation tasks into simpler, individual segmentations. MDIN comprises two fundamental components: Text-driven Sparse Queries (TSQ) and Multi-object Decoupling Optimization (MDO). TSQ generates sparse point cloud features distributed over key targets as the initialization for queries. Meanwhile, MDO is tasked with assigning each target in multi-object scenarios to different queries while maintaining their semantic consistency. To adapt to this new task, we build a new dataset, namely Multi3DRes. Our comprehensive evaluations on this dataset demonstrate substantial enhancements over existing models, thus charting a new path for intricate multi-object 3D scene comprehension. The benchmark and code are available at https://github.com/sosppxo/MDIN.