# Semantic Scene Completion via Integrating Instances and Scene in-the-Loop

arXiv: https://arxiv.org/abs/2104.03640 · HTML: https://mlsanity.com/paper/2104.03640/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 8 Apr 2021.

## Builds on (most cited first, 29 of 29)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [PCN: Point Completion Network](https://mlsanity.com/paper/1808.00671/) (2018)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018)
- [GRNet: Gridding Residual Network for Dense Point Cloud Completion](https://mlsanity.com/paper/2006.03761/) (2020)
- [PF-Net: Point Fractal Network for 3D Point Cloud Completion](https://mlsanity.com/paper/2003.00410/) (2020)
- [Anisotropic Convolutional Networks for 3D Semantic Scene Completion](https://mlsanity.com/paper/2004.02122/) (2020)
- [3D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Prior](https://mlsanity.com/paper/2003.14052/) (2020)
- [Cascaded Refinement Network for Point Cloud Completion](https://mlsanity.com/paper/2004.03327/) (2020)
- [Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images](https://mlsanity.com/paper/1511.02300/) (2015)
- [Point Cloud Completion by Skip-attention Network with Hierarchical Folding](https://mlsanity.com/paper/2005.03871/) (2020)
- [Efficient Semantic Scene Completion Network with Spatial Group Convolution](https://mlsanity.com/paper/1907.05091/) (2019)
- [RGBD Based Dimensional Decomposition Residual Network for 3D Semantic Scene Completion](https://mlsanity.com/paper/1903.00620/) (2019)
- [Two Stream 3D Semantic Scene Completion](https://mlsanity.com/paper/1804.03550/) (2018)
- [SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans](https://mlsanity.com/paper/2003.12622/) (2020)
- [Cascaded Context Pyramid for Full-Resolution 3D Semantic Scene Completion](https://mlsanity.com/paper/1908.00382/) (2019)
- [ForkNet: Multi-branch Volumetric Semantic Completion from a Single Depth Image](https://mlsanity.com/paper/1909.01106/) (2019)
- [RevealNet: Seeing Behind Objects in RGB-D Scans](https://mlsanity.com/paper/1904.12012/) (2019)
- [View-volume Network for Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1806.05361/) (2018)
- [SCFusion: Real-time Incremental Scene Reconstruction with Semantic Completion](https://mlsanity.com/paper/2010.13662/) (2020)
- [Semantic Scene Completion Combining Colour and Depth: preliminary experiments](https://mlsanity.com/paper/1802.04735/) (2018)

## Built on by (most cited first, 30 of 55)

- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.12251/) (2023)
- [OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception](https://mlsanity.com/paper/2303.03991/) (2023)
- [SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction](https://mlsanity.com/paper/2311.12754/) (2023)
- [Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models](https://mlsanity.com/paper/2207.11514/) (2022)
- [NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space](https://mlsanity.com/paper/2309.14616/) (2023)
- [AdaPoinTr: Diverse Point Cloud Completion with Adaptive Geometry-Aware Transformers](https://mlsanity.com/paper/2301.04545/) (2023)
- [POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images](https://mlsanity.com/paper/2401.09413/) (2024)
- [EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understanding](https://mlsanity.com/paper/2412.04380/) (2024)
- [Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillation](https://mlsanity.com/paper/2404.11958/) (2024)
- [PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness](https://mlsanity.com/paper/2312.02158/) (2023)
- [MotionSC: Data Set and Network for Real-Time Semantic Mapping in Dynamic Environments](https://mlsanity.com/paper/2203.07060/) (2022)
- [Occupancy as Set of Points](https://mlsanity.com/paper/2407.04049/) (2024)
- [RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radar](https://mlsanity.com/paper/2405.14014/) (2024)
- [Hierarchical Temporal Context Learning for Camera-based Semantic Scene Completion](https://mlsanity.com/paper/2407.02077/) (2024)
- [SSC-RS: Elevate LiDAR Semantic Scene Completion with Representation Separation and BEV Fusion](https://mlsanity.com/paper/2306.15349/) (2023)
- [S4C: Self-Supervised Semantic Scene Completion with Neural Fields](https://mlsanity.com/paper/2310.07522/) (2023)
- [Bridging Stereo Geometry and BEV Representation with Reliable Mutual Interaction for Semantic Scene Completion](https://mlsanity.com/paper/2303.13959/) (2023)
- [Learning a Structured Latent Space for Unsupervised Point Cloud Completion](https://mlsanity.com/paper/2203.15580/) (2022)
- [Learning Local Displacements for Point Cloud Completion](https://mlsanity.com/paper/2203.16600/) (2022)
- [HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction](https://mlsanity.com/paper/2408.09104/) (2024)
- [Feed-Forward SceneDINO for Unsupervised Semantic Scene Completion](https://mlsanity.com/paper/2507.06230/) (2025)
- [Hierarchical Dynamic Image Harmonization](https://mlsanity.com/paper/2211.08639/) (2022)
- [Semantic Scene Completion with Cleaner Self](https://mlsanity.com/paper/2303.09977/) (2023)
- [SLCF-Net: Sequential LiDAR-Camera Fusion for Semantic Scene Completion using a 3D Recurrent U-Net](https://mlsanity.com/paper/2403.08885/) (2024)
- [CasFusionNet: A Cascaded Network for Point Cloud Semantic Scene Completion by Dense Feature Fusion](https://mlsanity.com/paper/2211.13702/) (2022)
- [OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation](https://mlsanity.com/paper/2412.11183/) (2024)
- [M3Net: Multimodal Multi-task Learning for 3D Detection, Segmentation, and Occupancy Prediction in Autonomous Driving](https://mlsanity.com/paper/2503.18100/) (2025)
- [Data Augmented 3D Semantic Scene Completion with 2D Segmentation Priors](https://mlsanity.com/paper/2111.13309/) (2021)
- [Label-efficient Semantic Scene Completion with Scribble Annotations](https://mlsanity.com/paper/2405.15170/) (2024)

## Abstract (arXiv)

Semantic Scene Completion aims at reconstructing a complete 3D scene with precise voxel-wise semantics from a single-view depth or RGBD image. It is a crucial but challenging problem for indoor scene understanding. In this work, we present a novel framework named Scene-Instance-Scene Network (textitSISNet), which takes advantages of both instance and scene level semantic information. Our method is capable of inferring fine-grained shape details as well as nearby objects whose semantic categories are easily mixed-up. The key insight is that we decouple the instances from a coarsely completed semantic scene instead of a raw input image to guide the reconstruction of instances and the overall scene. SISNet conducts iterative scene-to-instance (SI) and instance-to-scene (IS) semantic completion. Specifically, the SI is able to encode objects' surrounding context for effectively decoupling instances from the scene and each instance could be voxelized into higher resolution to capture finer details. With IS, fine-grained instance information can be integrated back into the 3D scene and thus leads to more accurate semantic scene completion. Utilizing such an iterative mechanism, the scene and instance completion benefits each other to achieve higher completion accuracy. Extensively experiments show that our proposed method consistently outperforms state-of-the-art methods on both real NYU, NYUCAD and synthetic SUNCG-RGBD datasets. The code and the supplementary material will be available at urlhttps://github.com/yjcaimeow/SISNet.