mlsanityData to 14 Oct 2026

Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding

arXiv 2311.18482 ↗First posted 30 Nov 2023cs.CV cs.GRcited by 208

Not on the map. Why?

Abstract (arXiv)

Open-vocabulary querying in 3D space is challenging but essential for scene understanding tasks such as object localization and segmentation. Language-embedded scene representations have made progress by incorporating language features into 3D spaces. However, their efficacy heavily depends on neural networks that are resource-intensive in training and rendering. Although recent 3D Gaussians offer efficient and high-quality novel view synthesis, directly embedding language features in them leads to prohibitive memory usage and decreased performance. In this work, we introduce Language Embedded 3D Gaussians, a novel scene representation for open-vocabulary query tasks. Instead of embedding high-dimensional raw semantic features on 3D Gaussians, we propose a dedicated quantization scheme that drastically alleviates the memory requirement, and a novel embedding procedure that achieves smoother yet high accuracy query, countering the multi-view feature inconsistencies and the high-frequency inductive bias in point-based representations. Our comprehensive experiments show that our representation achieves the best visual quality and language querying accuracy across current language-embedded representations, while maintaining real-time rendering frame rates on a single desktop GPU.

Key facts. Cited by 208 papers in the map (data through 14 Oct 2026). Builds on 47 of its 64 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

47 of 64 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  10. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  11. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,613
  12. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
35 more
  1. 2022TensoRF: Tensorial Radiance Fieldscited by 1,280
  2. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,232
  3. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 964
  4. 2020Neural Sparse Voxel Fieldscited by 959
  5. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 861
  6. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 724
  7. 2022Language-driven Semantic Segmentationcited by 717
  8. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 671
  9. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  10. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  11. 2023LERF: Language Embedded Radiance Fieldscited by 543
  12. 2022Point-NeRF: Point-based Neural Radiance Fieldscited by 503
  13. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  14. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 415
  15. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  16. 2021Deep ViT Features as Dense Visual Descriptorscited by 330
  17. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  18. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  19. 2021Urban Radiance Fieldscited by 263
  20. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  21. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 202
  22. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 196
  23. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187
  24. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  25. 2022Panoptic Lifting for 3D Scene Understanding with Neural Fieldscited by 174
  26. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  27. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 160
  28. 2022Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentationcited by 152
  29. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  30. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  31. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  32. 2023Open-Vocabulary Point-Cloud Object Detection without 3D Annotationcited by 81
  33. 2022DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Imagescited by 69
  34. 2023Nerflets: Local Radiance Fields for Efficient Structure-Aware 3D Scene Representation from 2D Supervisioncited by 44
  35. 2023AutoRecon: Automated 3D Object Discovery and Reconstructioncited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 49; 2025: 84; 2026: 75.

208 papers.

Sort:

By area:

  1. 2024A Survey on 3D Gaussian Splattingcited by 224
  2. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 145
  3. 2024Recent Advances in 3D Gaussian Splattingcited by 131
  4. 2024RadSplat: Radiance Field-Informed Gaussian Splatting for Robust Real-Time Rendering with 900+ FPScited by 113
  5. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 108
  6. 20243D Gaussian as a New Era: A Surveycited by 104
  7. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  8. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 82
  9. 20243D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunitiescited by 78
  10. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 71
  11. 2024GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplanecited by 53
  12. 2025SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretrainingcited by 52
38 more
  1. 2025Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registrationcited by 51
  2. 2024Director3D: Real-world Camera Trajectory and 3D Scene Generation from Textcited by 50
  3. 2024Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editingcited by 45
  4. 2024SplatLoc: 3D Gaussian Splatting-based Visual Localization for Augmented Realitycited by 43
  5. 2024Gaga: Group Any Gaussians via 3D-aware Memory Bankcited by 41
  6. 2025LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPScited by 41
  7. 2025Deformable Beta Splattingcited by 38
  8. 2024LUDVIG: Learning-Free Uplifting of 2D Visual Features to Gaussian Splatting Scenescited by 37
  9. 2024InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perceptioncited by 35
  10. 20254D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Modelscited by 35
  11. 2024WE-GS: An In-the-wild Efficient 3D Gaussian Representation for Unconstrained Photo Collectionscited by 33
  12. 2024SuperGSeg: Open-Vocabulary 3D Segmentation with Structured Super-Gaussianscited by 33
  13. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  14. 20243D Vision-Language Gaussian Splattingcited by 31
  15. 2024GauStudio: A Modular Framework for 3D Gaussian Splatting and Beyondcited by 25
  16. 2024LM-Gaussian: Boost Sparse-view 3D Gaussian Splatting with Large Model Priorscited by 25
  17. 2024GaussianBeV: 3D Gaussian Representation meets Perception Models for BeV Segmentationcited by 20
  18. 2025OpenSplat3D: Open-Vocabulary 3D Instance Segmentation using Gaussian Splattingcited by 20
  19. 2025ReferSplat: Referring Segmentation in 3D Gaussian Splattingcited by 20
  20. 2025A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generationcited by 19
  21. 2024SLGaussian: Fast Language Gaussian Splatting in Sparse Viewscited by 18
  22. 2024GAGS: Granularity-Aware Feature Distillation for Language Gaussian Splattingcited by 18
  23. 2024PLGS: Robust Panoptic Lifting with 3D Gaussian Splattingcited by 17
  24. 2024LangSurf: Language-Embedded Surface Gaussians for 3D Scene Understandingcited by 17
  25. 2024CLIP-GS: CLIP-Informed Gaussian Splatting for View-Consistent 3D Indoor Semantic Understandingcited by 16
  26. 2024Occam's LGS: An Efficient Approach for Language Gaussian Splattingcited by 16
  27. 2024HybridGS: Decoupling Transients and Statics with 2D and 3D Gaussian Splattingcited by 16
  28. 2025PanoGS: Gaussian-based Panoptic Segmentation for 3D Open Vocabulary Scene Understandingcited by 16
  29. 2024GSemSplat: Generalizable Semantic 3D Gaussian Splatting from Uncalibrated Image Pairscited by 15
  30. 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 15
  31. 2025SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splattingcited by 15
  32. 2025UniForward: Unified 3D Scene and Semantic Field Reconstruction via Feed-Forward Gaussian Splatting from Only Sparse-View Imagescited by 14
  33. 2025SemanticSplat: Feed-Forward 3D Scene Understanding with Language-Aware Gaussian Fieldscited by 14
  34. 20243DGS-DET: Empower 3D Gaussian Splatting with Boundary Guidance and Box-Focused Sampling for Indoor 3D Object Detectioncited by 13
  35. 2024T-3DGS: Removing Transient Objects for 3D Scene Reconstructioncited by 13
  36. 2025Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile Gaussian Feature Fieldscited by 13
  37. 2024Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Spacecited by 12
  38. 20244-LEGS: 4D Language Embedded Gaussian Splattingcited by 12

50 most cited of 208 All 208 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond
  2. 2025CORE-3D: Context-aware Open-vocabulary Retrieval by Embeddings in 3D
  3. 2026GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generation
  4. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding
  5. 2026A Scene Language Model for Open-Vocabulary Scene Mapping
  6. 2026GoDeep: Annotation-Free Open-Vocabulary 3D Scene Understanding via Language-Space Lifting
  7. 2024TIGER: Text-Instructed 3D Gaussian Retrieval and Coherent Editing
  8. 2025Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding