# OccDepth: A Depth-Aware Method for 3D Semantic Scene Completion

arXiv: https://arxiv.org/abs/2302.13540 · HTML: https://mlsanity.com/paper/2302.13540/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 54 references that are in the map.
- First posted on arXiv 27 Feb 2023.

## Builds on (most cited first, 30 of 34)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [Categorical Depth Distribution Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.01100/) (2021)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion](https://mlsanity.com/paper/2012.03762/) (2020)
- [Unifying Voxel-based Representation with Transformer for 3D Object Detection](https://mlsanity.com/paper/2206.00630/) (2022)
- [S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Clouds](https://mlsanity.com/paper/2012.09242/) (2020)
- [SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans](https://mlsanity.com/paper/1912.00036/) (2019)
- [Semantic Scene Completion using Local Deep Implicit Functions on LiDAR Data](https://mlsanity.com/paper/2011.09141/) (2020)
- [3D Semantic Scene Completion: a Survey](https://mlsanity.com/paper/2103.07466/) (2021)
- [Panoptic 3D Scene Reconstruction From a Single RGB Image](https://mlsanity.com/paper/2111.02444/) (2021)
- [MonoDistill: Learning Spatial Features for Monocular 3D Object Detection](https://mlsanity.com/paper/2201.10830/) (2022)
- [Inter-Region Affinity Distillation for Road Marking Segmentation](https://mlsanity.com/paper/2004.05304/) (2020)
- [RGBD Based Dimensional Decomposition Residual Network for 3D Semantic Scene Completion](https://mlsanity.com/paper/1903.00620/) (2019)
- [Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation](https://mlsanity.com/paper/1903.04202/) (2019)
- [SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields](https://mlsanity.com/paper/2212.02501/) (2022)
- [Two Stream 3D Semantic Scene Completion](https://mlsanity.com/paper/1804.03550/) (2018)
- [Semantic Segmentation-assisted Scene Completion for LiDAR Point Clouds](https://mlsanity.com/paper/2109.11453/) (2021)
- [Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More](https://mlsanity.com/paper/1904.10167/) (2019)
- [Attention-based Multi-modal Fusion Network for Semantic Scene Completion](https://mlsanity.com/paper/2003.13910/) (2020)
- [Center3D: Center-based Monocular 3D Object Detection with Joint Depth Understanding](https://mlsanity.com/paper/2005.13423/) (2020)
- [SCFusion: Real-time Incremental Scene Reconstruction with Semantic Completion](https://mlsanity.com/paper/2010.13662/) (2020)

## Built on by (most cited first, 30 of 86)

- [Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving](https://mlsanity.com/paper/2304.14365/) (2023)
- [Scene as Occupancy](https://mlsanity.com/paper/2306.02851/) (2023)
- [GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2405.17429/) (2024)
- [RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervision](https://mlsanity.com/paper/2309.09502/) (2023)
- [PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentation](https://mlsanity.com/paper/2306.10013/) (2023)
- [COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Prediction](https://mlsanity.com/paper/2312.01919/) (2023)
- [Symphonize 3D Semantic Scene Completion with Contextual Instance Queries](https://mlsanity.com/paper/2306.15670/) (2023)
- [GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Prediction](https://mlsanity.com/paper/2412.04384/) (2024)
- [SemCity: Semantic Scene Generation with Triplane Diffusion](https://mlsanity.com/paper/2403.07773/) (2024)
- [Context and Geometry Aware Voxel Transformer for Semantic Scene Completion](https://mlsanity.com/paper/2405.13675/) (2024)
- [OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queries](https://mlsanity.com/paper/2312.03774/) (2023)
- [Camera-based 3D Semantic Scene Completion with Sparse Guidance Network](https://mlsanity.com/paper/2312.05752/) (2023)
- [Occupancy as Set of Points](https://mlsanity.com/paper/2407.04049/) (2024)
- [ViewFormer: Exploring Spatiotemporal Modeling for Multi-View 3D Occupancy Perception via View-Guided Transformers](https://mlsanity.com/paper/2405.04299/) (2024)
- [A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspective](https://mlsanity.com/paper/2405.05173/) (2024)
- [Learning Occupancy for Monocular 3D Object Detection](https://mlsanity.com/paper/2305.15694/) (2023)
- [UniWorld: Autonomous Driving Pre-training via World Models](https://mlsanity.com/paper/2308.07234/) (2023)
- [MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations](https://mlsanity.com/paper/2311.11762/) (2023)
- [Collaborative Semantic Occupancy Prediction with Hybrid Feature Fusion in Connected Automated Vehicles](https://mlsanity.com/paper/2402.07635/) (2024)
- [DepthSSC: Monocular 3D Semantic Scene Completion via Depth-Spatial Alignment and Voxel Adaptation](https://mlsanity.com/paper/2311.17084/) (2023)
- [Vision-based 3D occupancy prediction in autonomous driving: a review and outlook](https://mlsanity.com/paper/2405.02595/) (2024)
- [S4C: Self-Supervised Semantic Scene Completion with Neural Fields](https://mlsanity.com/paper/2310.07522/) (2023)
- [Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2303.01212/) (2023)
- [OVO: Open-Vocabulary Occupancy](https://mlsanity.com/paper/2305.16133/) (2023)
- [Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities](https://mlsanity.com/paper/2401.08045/) (2024)
- [CVT-Occ: Cost Volume Temporal Fusion for 3D Occupancy Prediction](https://mlsanity.com/paper/2409.13430/) (2024)
- [The RoboDrive Challenge: Drive Anytime Anywhere in Any Condition](https://mlsanity.com/paper/2405.08816/) (2024)
- [STCOcc: Sparse Spatial-Temporal Cascade Renovation for 3D Occupancy and Scene Flow Prediction](https://mlsanity.com/paper/2504.19749/) (2025)
- [OccFlowNet: Towards Self-supervised Occupancy Estimation via Differentiable Rendering and Occupancy Flow](https://mlsanity.com/paper/2402.12792/) (2024)
- [VEON: Vocabulary-Enhanced Occupancy Prediction](https://mlsanity.com/paper/2407.12294/) (2024)

## Abstract (arXiv)

3D Semantic Scene Completion (SSC) can provide dense geometric and semantic scene representations, which can be applied in the field of autonomous driving and robotic systems. It is challenging to estimate the complete geometry and semantics of a scene solely from visual images, and accurate depth information is crucial for restoring 3D geometry. In this paper, we propose the first stereo SSC method named OccDepth, which fully exploits implicit depth information from stereo images (or RGBD images) to help the recovery of 3D geometric structures. The Stereo Soft Feature Assignment (Stereo-SFA) module is proposed to better fuse 3D depth-aware features by implicitly learning the correlation between stereo images. In particular, when the input are RGBD image, a virtual stereo images can be generated through original RGB image and depth map. Besides, the Occupancy Aware Depth (OAD) module is used to obtain geometry-aware 3D features by knowledge distillation using pre-trained depth models. In addition, a reformed TartanAir benchmark, named SemanticTartanAir, is provided in this paper for further testing our OccDepth method on SSC task. Compared with the state-of-the-art RGB-inferred SSC method, extensive experiments on SemanticKITTI show that our OccDepth method achieves superior performance with improving +4.82% mIoU, of which +2.49% mIoU comes from stereo images and +2.33% mIoU comes from our proposed depth-aware method. Our code and trained models are available at https://github.com/megvii-research/OccDepth.