# Language and Geometry Grounded Sparse Voxel Representations for Holistic Scene Understanding

arXiv: https://arxiv.org/abs/2602.15734 · HTML: https://mlsanity.com/paper/2602.15734/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 38 references that are in the map.
- First posted on arXiv 17 Feb 2026.

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [Mip-Splatting: Alias-free 3D Gaussian Splatting](https://mlsanity.com/paper/2311.16493/) (2023)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS](https://mlsanity.com/paper/2311.17245/) (2023)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023)
- [Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.04803/) (2023)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](https://mlsanity.com/paper/2312.03203/) (2023)
- [ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding](https://mlsanity.com/paper/2212.05171/) (2022)
- [Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding](https://mlsanity.com/paper/2311.18482/) (2023)
- [HUGS: Human Gaussian Splats](https://mlsanity.com/paper/2311.17910/) (2023)
- [Weakly Supervised 3D Open-vocabulary Segmentation](https://mlsanity.com/paper/2305.14093/) (2023)
- [SparseGS: Sparse View Synthesis using 3D Gaussian Splatting](https://mlsanity.com/paper/2312.00206/) (2023)
- [FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding](https://mlsanity.com/paper/2401.01970/) (2024)
- [Open-Vocabulary Point-Cloud Object Detection without 3D Annotation](https://mlsanity.com/paper/2304.00788/) (2023)
- [LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPS](https://mlsanity.com/paper/2507.07136/) (2025)
- [Sparse Voxels Rasterization: Real-time High-fidelity Radiance Field Rendering](https://mlsanity.com/paper/2412.04459/) (2024)
- [3D Vision-Language Gaussian Splatting](https://mlsanity.com/paper/2410.07577/) (2024)
- [GAGS: Granularity-Aware Feature Distillation for Language Gaussian Splatting](https://mlsanity.com/paper/2412.13654/) (2024)

## Abstract (arXiv)

Existing 3D open-vocabulary scene understanding methods mostly emphasize distilling language features from 2D foundation models into 3D feature fields, but largely overlook the synergy among scene appearance, semantics, and geometry. As a result, scene understanding often deviates from the underlying geometric structure of scenes and becomes decoupled from the reconstruction process. In this work, we propose a novel approach that leverages language and geometry grounded sparse voxel representations to comprehensively model appearance, semantics, and geometry within a unified framework. Specifically, we use 3D sparse voxels as primitives and employ an appearance field, a density field, a feature field, and a confidence field to holistically represent a 3D scene. To promote synergy among the appearance, density, and feature fields, we construct a feature modulation module and distill language features from a 2D foundation model into our 3D scene model. In addition, we integrate geometric distillation into feature field distillation to transfer geometric knowledge from a geometry foundation model to our 3D scene representations via depth correlation regularization and pattern consistency regularization. These components work together to synergistically model the appearance, semantics, and geometry of the 3D scene within a unified framework. Extensive experiments demonstrate that our approach achieves superior overall performance compared with state-of-the-art methods in holistic scene understanding and reconstruction.