# Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding

arXiv: https://arxiv.org/abs/2311.18482 · HTML: https://mlsanity.com/paper/2311.18482/

## Key facts

- Cited by 208 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 64 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [Point-NeRF: Point-based Neural Radiance Fields](https://mlsanity.com/paper/2201.08845/) (2022)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023)
- [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023)

## Built on by (most cited first, 30 of 208)

- [A Survey on 3D Gaussian Splatting](https://mlsanity.com/paper/2401.03890/) (2024)
- [OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding](https://mlsanity.com/paper/2406.02058/) (2024)
- [Recent Advances in 3D Gaussian Splatting](https://mlsanity.com/paper/2403.11134/) (2024)
- [RadSplat: Radiance Field-Informed Gaussian Splatting for Robust Real-Time Rendering with 900+ FPS](https://mlsanity.com/paper/2403.13806/) (2024)
- [ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation](https://mlsanity.com/paper/2403.08321/) (2024)
- [3D Gaussian as a New Era: A Survey](https://mlsanity.com/paper/2402.07181/) (2024)
- [FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding](https://mlsanity.com/paper/2401.01970/) (2024)
- [Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting](https://mlsanity.com/paper/2403.15624/) (2024)
- [3D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunities](https://mlsanity.com/paper/2407.17418/) (2024)
- [Improving 2D Feature Representations by 3D-Aware Fine-Tuning](https://mlsanity.com/paper/2407.20229/) (2024)
- [GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplane](https://mlsanity.com/paper/2405.17596/) (2024)
- [SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretraining](https://mlsanity.com/paper/2503.18052/) (2025)
- [Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration](https://mlsanity.com/paper/2502.16652/) (2025)
- [Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text](https://mlsanity.com/paper/2406.17601/) (2024)
- [Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editing](https://mlsanity.com/paper/2404.01223/) (2024)
- [SplatLoc: 3D Gaussian Splatting-based Visual Localization for Augmented Reality](https://mlsanity.com/paper/2409.14067/) (2024)
- [Gaga: Group Any Gaussians via 3D-aware Memory Bank](https://mlsanity.com/paper/2404.07977/) (2024)
- [LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPS](https://mlsanity.com/paper/2507.07136/) (2025)
- [Deformable Beta Splatting](https://mlsanity.com/paper/2501.18630/) (2025)
- [LUDVIG: Learning-Free Uplifting of 2D Visual Features to Gaussian Splatting Scenes](https://mlsanity.com/paper/2410.14462/) (2024)
- [InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception](https://mlsanity.com/paper/2411.19235/) (2024)
- [4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models](https://mlsanity.com/paper/2503.10437/) (2025)
- [WE-GS: An In-the-wild Efficient 3D Gaussian Representation for Unconstrained Photo Collections](https://mlsanity.com/paper/2406.02407/) (2024)
- [SuperGSeg: Open-Vocabulary 3D Segmentation with Structured Super-Gaussians](https://mlsanity.com/paper/2412.10231/) (2024)
- [Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graph](https://mlsanity.com/paper/2406.07113/) (2024)
- [3D Vision-Language Gaussian Splatting](https://mlsanity.com/paper/2410.07577/) (2024)
- [GauStudio: A Modular Framework for 3D Gaussian Splatting and Beyond](https://mlsanity.com/paper/2403.19632/) (2024)
- [LM-Gaussian: Boost Sparse-view 3D Gaussian Splatting with Large Model Priors](https://mlsanity.com/paper/2409.03456/) (2024)
- [GaussianBeV: 3D Gaussian Representation meets Perception Models for BeV Segmentation](https://mlsanity.com/paper/2407.14108/) (2024)
- [OpenSplat3D: Open-Vocabulary 3D Instance Segmentation using Gaussian Splatting](https://mlsanity.com/paper/2506.07697/) (2025)

## Abstract (arXiv)

Open-vocabulary querying in 3D space is challenging but essential for scene understanding tasks such as object localization and segmentation. Language-embedded scene representations have made progress by incorporating language features into 3D spaces. However, their efficacy heavily depends on neural networks that are resource-intensive in training and rendering. Although recent 3D Gaussians offer efficient and high-quality novel view synthesis, directly embedding language features in them leads to prohibitive memory usage and decreased performance. In this work, we introduce Language Embedded 3D Gaussians, a novel scene representation for open-vocabulary query tasks. Instead of embedding high-dimensional raw semantic features on 3D Gaussians, we propose a dedicated quantization scheme that drastically alleviates the memory requirement, and a novel embedding procedure that achieves smoother yet high accuracy query, countering the multi-view feature inconsistencies and the high-frequency inductive bias in point-based representations. Our comprehensive experiments show that our representation achieves the best visual quality and language querying accuracy across current language-embedded representations, while maintaining real-time rendering frame rates on a single desktop GPU.