# EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understanding

arXiv: https://arxiv.org/abs/2412.04380 · HTML: https://mlsanity.com/paper/2412.04380/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 60 references that are in the map.
- First posted on arXiv 5 Dec 2024.

## Builds on (most cited first, 30 of 54)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2302.07817/) (2023)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2303.09551/) (2023)
- [VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.12251/) (2023)
- [GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting](https://mlsanity.com/paper/2311.11700/) (2023)
- [OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving](https://mlsanity.com/paper/2311.16038/) (2023)
- [OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception](https://mlsanity.com/paper/2303.03991/) (2023)
- [Scene as Occupancy](https://mlsanity.com/paper/2306.02851/) (2023)
- [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022)
- [FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformation](https://mlsanity.com/paper/2307.01492/) (2023)
- [3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos](https://mlsanity.com/paper/2403.01444/) (2024)
- [GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2405.17429/) (2024)
- [Gaussian-SLAM: Photo-realistic Dense SLAM with Gaussian Splatting](https://mlsanity.com/paper/2312.10070/) (2023)
- [EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI](https://mlsanity.com/paper/2312.16170/) (2023)
- [GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud](https://mlsanity.com/paper/1812.03320/) (2018)
- [SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction](https://mlsanity.com/paper/2311.12754/) (2023)
- [3D Geometry-aware Deformable Gaussian Splatting for Dynamic View Synthesis](https://mlsanity.com/paper/2404.06270/) (2024)
- [ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans](https://mlsanity.com/paper/1712.10215/) (2017)
- [GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation](https://mlsanity.com/paper/2403.12365/) (2024)
- [FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection](https://mlsanity.com/paper/2112.00322/) (2021)
- [NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space](https://mlsanity.com/paper/2309.14616/) (2023)

## Built on by (most cited first, 30 of 45)

- [Streaming 4D Visual Geometry Transformer](https://mlsanity.com/paper/2507.11539/) (2025)
- [InfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streams](https://mlsanity.com/paper/2601.02281/) (2026)
- [Doe-1: Closed-Loop Autonomous Driving with Large World Model](https://mlsanity.com/paper/2412.09627/) (2024)
- [ORV: 4D Occupancy-centric Robot Video Generation](https://mlsanity.com/paper/2506.03079/) (2025)
- [VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model](https://mlsanity.com/paper/2603.12655/) (2026)
- [Generalizing Visual Geometry Priors to Sparse Gaussian Occupancy Prediction](https://mlsanity.com/paper/2602.21552/) (2026)
- [SplatSSC: Decoupled Depth-Guided Gaussian Splatting for Semantic Scene Completion](https://mlsanity.com/paper/2508.02261/) (2025)
- [SliceOcc: Indoor 3D Semantic Occupancy Prediction with Vertical Slice Representation](https://mlsanity.com/paper/2501.16684/) (2025)
- [Occupancy World Model for Robots](https://mlsanity.com/paper/2505.05512/) (2025)
- [OneOcc: Semantic Occupancy Prediction for Legged Robots with a Single Panoramic Camera](https://mlsanity.com/paper/2511.03571/) (2025)
- [Humanoid Occupancy: Enabling A Generalized Multimodal Occupancy Perception System on Humanoid Robots](https://mlsanity.com/paper/2507.20217/) (2025)
- [Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes](https://mlsanity.com/paper/2602.22667/) (2026)
- [Panoramic Multimodal Semantic Occupancy Prediction for Quadruped Robots](https://mlsanity.com/paper/2603.13108/) (2026)
- [SM4RT: Learning Structured Motion Geometry for 4D Reconstruction](https://mlsanity.com/paper/2607.22534/) (2026)
- [Terra: Explorable Native 3D World Model with Point Latents](https://mlsanity.com/paper/2510.14977/) (2025)
- [EA3D: Online Open-World 3D Object Extraction from Streaming Videos](https://mlsanity.com/paper/2510.25146/) (2025)
- [Enhancing Indoor Occupancy Prediction via Sparse Query-Based Multi-Level Consistent Knowledge Distillation](https://mlsanity.com/paper/2602.02318/) (2026)
- [OmniIndoor3D: Comprehensive Indoor 3D Reconstruction](https://mlsanity.com/paper/2505.20610/) (2025)
- [Unleashing Semantic and Geometric Priors for 3D Scene Completion](https://mlsanity.com/paper/2508.13601/) (2025)
- [TGSFormer: Scalable Temporal Gaussian Splatting for Embodied Semantic Scene Completion](https://mlsanity.com/paper/2512.00300/) (2025)
- [O3N: Omnidirectional Open-Vocabulary Occupancy Prediction for Embodied Intelligent Robotics](https://mlsanity.com/paper/2603.12144/) (2026)
- [FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction](https://mlsanity.com/paper/2604.28115/) (2026)
- [YouTube-Occ: Learning Indoor 3D Semantic Occupancy Prediction from YouTube Videos](https://mlsanity.com/paper/2506.18266/) (2025)
- [IVGT: Implicit Visual Geometry Transformer for Neural Scene Representation](https://mlsanity.com/paper/2605.16258/) (2026)
- [VEOcc: Voxel-Centric Online Semantic Occupancy Prediction For Embodied Scene Understanding](https://mlsanity.com/paper/2605.25059/) (2026)
- [FLM-Occ: Feed-forward Likelihood Maximization for Efficient Indoor Occupancy Prediction](https://mlsanity.com/paper/2606.21373/) (2026)
- [Humanoid-OmniOcc: Stereo-Based Full-View Occupancy Dataset for Embodied AI](https://mlsanity.com/paper/2606.22971/) (2026)
- [Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images](https://mlsanity.com/paper/2607.01633/) (2026)
- [Efficient and High-Quality Depth Estimation via Pixel-Space Diffusion with Linear Attention](https://mlsanity.com/paper/2608.30129/) (2026)
- [Rebenchmarking Unsupervised Monocular 3D Occupancy Prediction](https://mlsanity.com/paper/2602.06488/) (2026)

## Abstract (arXiv)

3D occupancy prediction provides a comprehensive description of the surrounding scenes and has become an essential task for 3D perception. Most existing methods focus on offline perception from one or a few views and cannot be applied to embodied agents that demand to gradually perceive the scene through progressive embodied exploration. In this paper, we formulate an embodied 3D occupancy prediction task to target this practical scenario and propose a Gaussian-based EmbodiedOcc framework to accomplish it. We initialize the global scene with uniform 3D semantic Gaussians and progressively update local regions observed by the embodied agent. For each update, we extract semantic and structural features from the observed image and efficiently incorporate them via deformable cross-attention to refine the regional Gaussians. Finally, we employ Gaussian-to-voxel splatting to obtain the global 3D occupancy from the updated 3D Gaussians. Our EmbodiedOcc assumes an unknown (i.e., uniformly distributed) environment and maintains an explicit global memory of it with 3D Gaussians. It gradually gains knowledge through the local refinement of regional Gaussians, which is consistent with how humans understand new scenes through embodied exploration. We reorganize an EmbodiedOcc-ScanNet benchmark based on local annotations to facilitate the evaluation of the embodied 3D occupancy prediction task. Our EmbodiedOcc outperforms existing methods by a large margin and accomplishes the embodied occupancy prediction with high accuracy and efficiency. Code: https://github.com/YkiWu/EmbodiedOcc.