# EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understanding

arXiv: https://arxiv.org/abs/2603.04254 · HTML: https://mlsanity.com/paper/2603.04254/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 82 of its 89 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Builds on (most cited first, 30 of 82)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction](https://mlsanity.com/paper/2312.12337/) (2023)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images](https://mlsanity.com/paper/2403.14627/) (2024)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021)
- [Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2004.02857/) (2020)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [Gaussian Splatting SLAM](https://mlsanity.com/paper/2312.06741/) (2023)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](https://mlsanity.com/paper/2312.03203/) (2023)
- [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023)
- [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023)
- [Language-Grounded Indoor 3D Semantic Segmentation in the Wild](https://mlsanity.com/paper/2204.07761/) (2022)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding](https://mlsanity.com/paper/2311.18482/) (2023)
- [PLA: Language-Driven Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2211.16312/) (2022)
- [Segment Any 3D Gaussians](https://mlsanity.com/paper/2312.00860/) (2023)

## Built on by (most cited first, 1 of 1)

- [DSG: Dynamic 3D Scene Graph Construction for Embodied Agents in Changing Indoor Environments](https://mlsanity.com/paper/2609.00619/) (2026)

## Abstract (arXiv)

Understanding a 3D scene immediately with its exploration is essential for embodied tasks, where an agent must construct and comprehend the 3D scene in an online and nearly real-time manner. In this study, we propose EmbodiedSplat, an online feed-forward 3DGS for open-vocabulary scene understanding that enables simultaneous online 3D reconstruction and 3D semantic understanding from the streaming images. Unlike existing open-vocabulary 3DGS methods which are typically restricted to either offline or per-scene optimization setting, our objectives are two-fold: 1) Reconstructs the semantic-embedded 3DGS of the entire scene from over 300 streaming images in an online manner. 2) Highly generalizable to novel scenes with feed-forward design and supports nearly real-time 3D semantic reconstruction when combined with real-time 2D models. To achieve these objectives, we propose an Online Sparse Coefficients Field with a CLIP Global Codebook where it binds the 2D CLIP embeddings to each 3D Gaussian while minimizing memory consumption and preserving the full semantic generalizability of CLIP. Furthermore, we generate 3D geometric-aware CLIP features by aggregating the partial point cloud of 3DGS through 3D U-Net to compensate the 3D geometric prior to 2D-oriented language embeddings. Extensive experiments on diverse indoor datasets, including ScanNet, ScanNet++, and Replica, demonstrate both the effectiveness and efficiency of our method. Check out our project page in https://0nandon.github.io/EmbodiedSplat/.