# CAGS: Open-Vocabulary 3D Scene Understanding with Context-Aware Gaussian Splatting

arXiv: https://arxiv.org/abs/2504.11893 · HTML: https://mlsanity.com/paper/2504.11893/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 16 Apr 2025.

## Builds on (most cited first, 28 of 28)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023)
- [SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM](https://mlsanity.com/paper/2312.02126/) (2023)
- [Compact 3D Gaussian Representation for Radiance Field](https://mlsanity.com/paper/2311.13681/) (2023)
- [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](https://mlsanity.com/paper/2312.03203/) (2023)
- [Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis](https://mlsanity.com/paper/2312.16812/) (2023)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Differentiable Surface Splatting for Point-based Geometry Processing](https://mlsanity.com/paper/1906.04173/) (2019)
- [Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding](https://mlsanity.com/paper/2311.18482/) (2023)
- [ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding](https://mlsanity.com/paper/2305.08275/) (2023)
- [PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning](https://mlsanity.com/paper/2211.11682/) (2022)
- [Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models](https://mlsanity.com/paper/2312.13763/) (2023)
- [OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding](https://mlsanity.com/paper/2406.02058/) (2024)
- [Click-Gaussian: Interactive Segmentation to Any 3D Gaussians](https://mlsanity.com/paper/2407.11793/) (2024)
- [Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splatting](https://mlsanity.com/paper/2503.14029/) (2025)
- [Correspondence-Guided SfM-Free 3D Gaussian Splatting for NVS](https://mlsanity.com/paper/2408.08723/) (2024)
- [Uncertainty-guided Optimal Transport in Depth Supervised Sparse-View 3D Gaussian](https://mlsanity.com/paper/2405.19657/) (2024)

## Built on by (most cited first, 11 of 11)

- [A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generation](https://mlsanity.com/paper/2508.09977/) (2025)
- [ExtrinSplat: Decoupling Geometry and Semantics for Open-Vocabulary Understanding in 3D Gaussian Splatting](https://mlsanity.com/paper/2509.22225/) (2025)
- [ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting](https://mlsanity.com/paper/2607.18801/) (2026)
- [ProFuse: Efficient Cross-View Context Fusion for Open-Vocabulary 3D Gaussian Splatting](https://mlsanity.com/paper/2601.04754/) (2026)
- [Split&Splat: Zero-Shot Panoptic Segmentation via Explicit Instance Modeling and 3D Gaussian Splatting](https://mlsanity.com/paper/2602.03809/) (2026)
- [TrackRef3D: Multi-View Consistent Track-then-Label for Open-World Referring Segmentation in 3D Gaussian Splatting](https://mlsanity.com/paper/2605.26576/) (2026)
- [OGGSplat: Open Gaussian Growing for Generalizable Reconstruction with Expanded Field-of-View](https://mlsanity.com/paper/2506.05204/) (2025)
- [LEGO: Leveled Language Gaussian Splatting](https://mlsanity.com/paper/2608.10057/) (2026)
- [Beyond Similarity Matching: Structured Reasoning for Open-Vocabulary Referring Segmentation in 3DGS](https://mlsanity.com/paper/2608.16103/) (2026)
- [NormLift: From Lifted Features To Semantic Reliability In 3D Gaussian Splatting](https://mlsanity.com/paper/2609.18898/) (2026)
- [GaussianDS: Depth-supervised Semantic Gaussian Splatting for Scene Understanding](https://mlsanity.com/paper/2609.27850/) (2026)

## Abstract (arXiv)

Open-vocabulary 3D scene understanding is crucial for applications requiring natural language-driven spatial interpretation, such as robotics and augmented reality. While 3D Gaussian Splatting (3DGS) offers a powerful representation for scene reconstruction, integrating it with open-vocabulary frameworks reveals a key challenge: cross-view granularity inconsistency. This issue, stemming from 2D segmentation methods like SAM, results in inconsistent object segmentations across views (e.g., a "coffee set" segmented as a single entity in one view but as "cup + coffee + spoon" in another). Existing 3DGS-based methods often rely on isolated per-Gaussian feature learning, neglecting the spatial context needed for cohesive object reasoning, leading to fragmented representations. We propose Context-Aware Gaussian Splatting (CAGS), a novel framework that addresses this challenge by incorporating spatial context into 3DGS. CAGS constructs local graphs to propagate contextual features across Gaussians, reducing noise from inconsistent granularity, employs mask-centric contrastive learning to smooth SAM-derived features across views, and leverages a precomputation strategy to reduce computational cost by precomputing neighborhood relationships, enabling efficient training in large-scale scenes. By integrating spatial context, CAGS significantly improves 3D instance segmentation and reduces fragmentation errors on datasets like LERF-OVS and ScanNet, enabling robust language-guided 3D scene understanding.