# Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images

arXiv: https://arxiv.org/abs/2607.01633 · HTML: https://mlsanity.com/paper/2607.01633/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 88 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Builds on (most cited first, 30 of 74)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction](https://mlsanity.com/paper/2312.12337/) (2023)
- [MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images](https://mlsanity.com/paper/2403.14627/) (2024)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2302.07817/) (2023)
- [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023)
- [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](https://mlsanity.com/paper/2312.03203/) (2023)
- [Splatter Image: Ultra-Fast Single-View 3D Reconstruction](https://mlsanity.com/paper/2312.13150/) (2023)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving](https://mlsanity.com/paper/2304.14365/) (2023)
- [SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2303.09551/) (2023)
- [MapAnything: Universal Feed-Forward Metric 3D Reconstruction](https://mlsanity.com/paper/2509.13414/) (2025)
- [VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.12251/) (2023)
- [OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2304.05316/) (2023)
- [No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images](https://mlsanity.com/paper/2410.24207/) (2024)
- [AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Views](https://mlsanity.com/paper/2505.23716/) (2025)
- [OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception](https://mlsanity.com/paper/2303.03991/) (2023)

## Built on by (most cited first, 1 of 1)

- [VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction](https://mlsanity.com/paper/2607.18078/) (2026)

## Abstract (arXiv)

Comprehensive 3D scene understanding from sparse, unposed images requires a model to recover renderable geometry, open-vocabulary semantics, and free/occupied 3D space without relying on external camera calibration. Recent feed-forward Gaussian methods improve pose-free reconstruction and semantic rendering, but their Gaussian primitives are mainly optimized through image-space objectives and remain weakly constrained in unobserved regions. We propose textitCOVScene, a pose-free semantic Gaussian framework that couples renderable Gaussian primitives with a dense semantic occupancy field through differentiable volumetric lifting. Instead of converting Gaussians to voxels only at evaluation time, COVScene lifts the predicted semantic Gaussians inside the training computation graph, so volumetric regularization provides gradients to Gaussian opacity, geometry, and semantic features. The framework combines a semantic-aware Geometry Transformer, multi-task Gaussian decoding, geometric foundation distillation, and occupancy entropy regularization to support novel view synthesis, open-vocabulary semantic querying, and semantic occupancy prediction within a single representation. Experiments on ScanNet and ScanNet++ show that COVScene maintains competitive rendering quality, improves open-vocabulary segmentation, and achieves stronger semantic occupancy prediction than the self-supervised baseline without direct voxel-level supervision.