# Semantics-Controlled Gaussian Splatting for Outdoor Scene Reconstruction and Rendering in Virtual Reality arXiv: https://arxiv.org/abs/2409.15959 ยท HTML: https://mlsanity.com/paper/2409.15959/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 26 of its 63 references that are in the map. - First posted on arXiv 24 Sep 2024. ## Builds on (most cited first, 26 of 26) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023) - [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022) - [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021) - [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020) - [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019) - [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023) - [Neuralangelo: High-Fidelity Neural Surface Reconstruction](https://mlsanity.com/paper/2306.03092/) (2023) - [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023) - [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023) - [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021) - [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](https://mlsanity.com/paper/2312.03203/) (2023) - [Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering](https://mlsanity.com/paper/2109.01847/) (2021) - [A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets](https://mlsanity.com/paper/2406.12080/) (2024) - [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023) - [VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality](https://mlsanity.com/paper/2401.16663/) (2024) - [Weakly Supervised 3D Open-vocabulary Segmentation](https://mlsanity.com/paper/2305.14093/) (2023) - [LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent](https://mlsanity.com/paper/2309.12311/) (2023) - [VR-NeRF: High-Fidelity Virtualized Walkable Spaces](https://mlsanity.com/paper/2311.02542/) (2023) - [FoV-NeRF: Foveated Neural Radiance Fields for Virtual Reality](https://mlsanity.com/paper/2103.16365/) (2021) - [360MonoDepth: High-Resolution 360deg Monocular Depth Estimation](https://mlsanity.com/paper/2111.15669/) (2021) - [Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databases](https://mlsanity.com/paper/2403.09675/) (2024) - [RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Rendering](https://mlsanity.com/paper/2212.01120/) (2022) - [Contrastive Gaussian Clustering: Weakly Supervised 3D Scene Segmentation](https://mlsanity.com/paper/2404.12784/) (2024) - [Factorized and Controllable Neural Re-Rendering of Outdoor Scene for Photo Extrapolation](https://mlsanity.com/paper/2207.06899/) (2022) - [NeRFtrinsic Four: An End-To-End Trainable NeRF Jointly Optimizing Diverse Intrinsic and Extrinsic Camera Parameters](https://mlsanity.com/paper/2303.09412/) (2023) ## Built on by (most cited first, 10 of 10) - [GradiSeg: Gradient-Guided Gaussian Segmentation with Enhanced 3D Boundary Precision](https://mlsanity.com/paper/2412.00392/) (2024) - [HoGS: Unified Near and Far Object Reconstruction via Homogeneous Gaussian Splatting](https://mlsanity.com/paper/2503.19232/) (2025) - [4DLangVGGT: 4D Language-Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2512.05060/) (2025) - [Consistent Instance Field for Dynamic Scene Understanding](https://mlsanity.com/paper/2512.14126/) (2025) - [CoRe-GS: Coarse-to-Refined Gaussian Splatting with Semantic Object Focus](https://mlsanity.com/paper/2509.04859/) (2025) - [DynamicTree: Interactive Real Tree Animation via Sparse Voxel Spectrum](https://mlsanity.com/paper/2510.22213/) (2025) - [Hybrid Foveated Path Tracing with Peripheral Gaussians for Immersive Anatomy](https://mlsanity.com/paper/2601.22026/) (2026) - [Abstract Rendering: Computing All that is Seen in Gaussian Splat Scenes](https://mlsanity.com/paper/2503.00308/) (2025) - [SAGE: Semantic-Driven Adaptive Gaussian Splatting in Extended Reality](https://mlsanity.com/paper/2503.16747/) (2025) - [Cross-Temporal 3D Gaussian Splatting for Sparse-View Guided Scene Update](https://mlsanity.com/paper/2512.00534/) (2025) ## Abstract (arXiv) Advancements in 3D rendering like Gaussian Splatting (GS) allow novel view synthesis and real-time rendering in virtual reality (VR). However, GS-created 3D environments are often difficult to edit. For scene enhancement or to incorporate 3D assets, segmenting Gaussians by class is essential. Existing segmentation approaches are typically limited to certain types of scenes, e.g., ''circular'' scenes, to determine clear object boundaries. However, this method is ineffective when removing large objects in non-''circling'' scenes such as large outdoor scenes. We propose Semantics-Controlled GS (SCGS), a segmentation-driven GS approach, enabling the separation of large scene parts in uncontrolled, natural environments. SCGS allows scene editing and the extraction of scene parts for VR. Additionally, we introduce a challenging outdoor dataset, overcoming the ''circling'' setup. We outperform the state-of-the-art in visual quality on our dataset and in segmentation quality on the 3D-OVS dataset. We conducted an exploratory user study, comparing a 360-video, plain GS, and SCGS in VR with a fixed viewpoint. In our subsequent main study, users were allowed to move freely, evaluating plain GS and SCGS. Our main study results show that participants clearly prefer SCGS over plain GS. We overall present an innovative approach that surpasses the state-of-the-art both technically and in user experience.