# DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting

arXiv: https://arxiv.org/abs/2412.10972 · HTML: https://mlsanity.com/paper/2412.10972/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 14 Dec 2024.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](https://mlsanity.com/paper/2312.03203/) (2023)
- [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023)
- [Segment Any 3D Gaussians](https://mlsanity.com/paper/2312.00860/) (2023)
- [SAM3D: Segment Anything in 3D Scenes](https://mlsanity.com/paper/2306.03908/) (2023)
- [Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation](https://mlsanity.com/paper/2203.15224/) (2022)
- [SAI3D: Segment Any Instance in 3D Scenes](https://mlsanity.com/paper/2312.11557/) (2023)
- [Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting](https://mlsanity.com/paper/2403.15624/) (2024)
- [OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views](https://mlsanity.com/paper/2404.03650/) (2024)
- [Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels](https://mlsanity.com/paper/2312.17232/) (2023)
- [Splat-Nav: Safe Real-Time Robot Navigation in Gaussian Splatting Maps](https://mlsanity.com/paper/2403.02751/) (2024)
- [GaussNav: Gaussian Splatting for Visual Navigation](https://mlsanity.com/paper/2403.11625/) (2024)
- [Open-Vocabulary Semantic Segmentation with Decoupled One-Pass Network](https://mlsanity.com/paper/2304.01198/) (2023)
- [UnScene3D: Unsupervised 3D Instance Segmentation for Indoor Scenes](https://mlsanity.com/paper/2303.14541/) (2023)
- [SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decomposition](https://mlsanity.com/paper/2401.17857/) (2024)

## Built on by (most cited first, 4 of 4)

- [OVI-MAP:Open-Vocabulary Instance-Semantic Mapping](https://mlsanity.com/paper/2603.26541/) (2026)
- [G2P: Gaussian-to-Point Attribute Alignment for Boundary-Aware 3D Segmentation](https://mlsanity.com/paper/2601.03510/) (2026)
- [Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds](https://mlsanity.com/paper/2608.00463/) (2026)
- [DIGS: Distribution-Informed Gaussian Splatting for Training-Free Open-Vocabulary 3D Segmentation](https://mlsanity.com/paper/2610.16177/) (2026)

## Abstract (arXiv)

Open-set 3D segmentation represents a major point of interest for multiple downstream robotics and augmented/virtual reality applications. We present a decoupled 3D segmentation pipeline to ensure modularity and adaptability to novel 3D representations as well as semantic segmentation foundation models. We first reconstruct a scene with 3D Gaussians and learn class-agnostic features through contrastive supervision from a 2D instance proposal network. These 3D features are then clustered to form coarse object- or part-level masks. Finally, we match each 3D cluster to class-aware masks predicted by a 2D open-vocabulary segmentation model, assigning semantic labels without retraining the 3D representation. Our decoupled design (1) provides a plug-and-play interface for swapping different 2D or 3D modules, (2) ensures multi-object instance segmentation at no extra cost, and (3) leverages rich 3D geometry for robust scene understanding. We evaluate on synthetic and real-world indoor datasets, demonstrating improved performance over comparable NeRF-based pipelines on mIoU and mAcc, particularly for challenging or long-tail classes. We also show how varying the 2D backbone affects the final segmentation, highlighting the modularity of our framework. These results confirm that decoupling 3D mask proposal and semantic classification can deliver flexible, efficient, and open-vocabulary 3D segmentation.