mlsanityData to 7 Oct 2026

Semantic Scene Completion via Integrating Instances and Scene in-the-Loop

arXiv 2104.03640 ↗First posted 8 Apr 2021cs.CVcited by 54

Not on the map. Why?

Abstract (arXiv)

Semantic Scene Completion aims at reconstructing a complete 3D scene with precise voxel-wise semantics from a single-view depth or RGBD image. It is a crucial but challenging problem for indoor scene understanding. In this work, we present a novel framework named Scene-Instance-Scene Network (textitSISNet), which takes advantages of both instance and scene level semantic information. Our method is capable of inferring fine-grained shape details as well as nearby objects whose semantic categories are easily mixed-up. The key insight is that we decouple the instances from a coarsely completed semantic scene instead of a raw input image to guide the reconstruction of instances and the overall scene. SISNet conducts iterative scene-to-instance (SI) and instance-to-scene (IS) semantic completion. Specifically, the SI is able to encode objects' surrounding context for effectively decoupling instances from the scene and each instance could be voxelized into higher resolution to capture finer details. With IS, fine-grained instance information can be integrated back into the 3D scene and thus leads to more accurate semantic scene completion. Utilizing such an iterative mechanism, the scene and instance completion benefits each other to achieve higher completion accuracy. Extensively experiments show that our proposed method consistently outperforms state-of-the-art methods on both real NYU, NYUCAD and synthetic SUNCG-RGBD datasets. The code and the supplementary material will be available at urlhttps://github.com/yjcaimeow/SISNet.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 8 Apr 2021.

Builds on

29 of 44 references.

References · 29

  1. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  3. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  4. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  5. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  6. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
  7. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  8. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  9. 20203DSSD: Point-based 3D Single Stage Object Detectorcited by 440
  10. 2018PCN: Point Completion Networkcited by 382
  11. 2019From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Networkcited by 366
  12. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 216
17 more
  1. 2020GRNet: Gridding Residual Network for Dense Point Cloud Completioncited by 174
  2. 2020PF-Net: Point Fractal Network for 3D Point Cloud Completioncited by 146
  3. 2020Anisotropic Convolutional Networks for 3D Semantic Scene Completioncited by 126
  4. 20203D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Priorcited by 124
  5. 2020Cascaded Refinement Network for Point Cloud Completioncited by 112
  6. 2015Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Imagescited by 111
  7. 2020Point Cloud Completion by Skip-attention Network with Hierarchical Foldingcited by 106
  8. 2019Efficient Semantic Scene Completion Network with Spatial Group Convolutioncited by 66
  9. 2019RGBD Based Dimensional Decomposition Residual Network for 3D Semantic Scene Completioncited by 54
  10. 2018Two Stream 3D Semantic Scene Completioncited by 48
  11. 2020SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scanscited by 44
  12. 2019Cascaded Context Pyramid for Full-Resolution 3D Semantic Scene Completioncited by 43
  13. 2019ForkNet: Multi-branch Volumetric Semantic Completion from a Single Depth Imagecited by 31
  14. 2019RevealNet: Seeing Behind Objects in RGB-D Scanscited by 30
  15. 2018View-volume Network for Semantic Scene Completion from a Single Depth Imagecited by 26
  16. 2020SCFusion: Real-time Incremental Scene Reconstruction with Semantic Completioncited by 25
  17. 2018Semantic Scene Completion Combining Colour and Depth: preliminary experimentscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 15; 2024: 19; 2025: 10; 2026: 2.

54 papers.

Sort:

By area:

  1. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 357
  2. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 306
  3. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 218
  4. 2023SelfOcc: Self-Supervised Vision-Based 3D Occupancy Predictioncited by 133
  5. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  6. 2023NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Spacecited by 88
  7. 2023AdaPoinTr: Diverse Point Cloud Completion with Adaptive Geometry-Aware Transformerscited by 65
  8. 2024POP-3D: Open-Vocabulary 3D Occupancy Prediction from Imagescited by 52
  9. 2024Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillationcited by 43
  10. 2024EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understandingcited by 43
  11. 2023PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awarenesscited by 42
  12. 2022MotionSC: Data Set and Network for Real-Time Semantic Mapping in Dynamic Environmentscited by 40
38 more
  1. 2024Occupancy as Set of Pointscited by 37
  2. 2024RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radarcited by 32
  3. 2024Hierarchical Temporal Context Learning for Camera-based Semantic Scene Completioncited by 32
  4. 2023SSC-RS: Elevate LiDAR Semantic Scene Completion with Representation Separation and BEV Fusioncited by 30
  5. 2023S4C: Self-Supervised Semantic Scene Completion with Neural Fieldscited by 27
  6. 2023Bridging Stereo Geometry and BEV Representation with Reliable Mutual Interaction for Semantic Scene Completioncited by 21
  7. 2022Learning a Structured Latent Space for Unsupervised Point Cloud Completioncited by 17
  8. 2022Learning Local Displacements for Point Cloud Completioncited by 16
  9. 2022Hierarchical Dynamic Image Harmonizationcited by 12
  10. 2023Semantic Scene Completion with Cleaner Selfcited by 12
  11. 2024SLCF-Net: Sequential LiDAR-Camera Fusion for Semantic Scene Completion using a 3D Recurrent U-Netcited by 12
  12. 2024HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Predictioncited by 12
  13. 2022CasFusionNet: A Cascaded Network for Point Cloud Semantic Scene Completion by Dense Feature Fusioncited by 11
  14. 2025Feed-Forward SceneDINO for Unsupervised Semantic Scene Completioncited by 11
  15. 2024OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generationcited by 10
  16. 2025M3Net: Multimodal Multi-task Learning for 3D Detection, Segmentation, and Occupancy Prediction in Autonomous Drivingcited by 10
  17. 2021Data Augmented 3D Semantic Scene Completion with 2D Segmentation Priorscited by 9
  18. 2024Label-efficient Semantic Scene Completion with Scribble Annotationscited by 9
  19. 2025Skip Mamba Diffusion for Monocular 3D Semantic Scene Completioncited by 9
  20. 2025PointSea: Point Cloud Completion via Self-structure Augmentationcited by 9
  21. 2024Hierarchical Context Alignment with Disentangled Geometric and Temporal Modeling for Semantic Occupancy Predictioncited by 8
  22. 2023SSR-2D: Semantic 3D Scene Reconstruction from 2D Imagescited by 7
  23. 2023Registered and Segmented Deformable Object Reconstruction from a Single View Point Cloudcited by 6
  24. 2023Semantic Complete Scene Forecasting from a 4D Dynamic Point Cloud Sequencecited by 6
  25. 2024Boosting Self-Supervision for Single-View Scene Completion via Knowledge Distillationcited by 6
  26. 2024ReliOcc: Towards Reliable Semantic Occupancy Prediction via Uncertainty Learningcited by 6
  27. 2024ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Drivingcited by 6
  28. 2025VLScene: Vision-Language Guidance Distillation for Camera-Based 3D Semantic Scene Completioncited by 6
  29. 2023CVSformer: Cross-View Synthesis Transformer for Semantic Scene Completioncited by 4
  30. 2024Unleashing Network Potentials for Semantic Scene Completioncited by 4
  31. 2024Deep Learning for 3D Point Cloud Enhancement: A Surveycited by 4
  32. 2025IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposalscited by 3
  33. 2025CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generationcited by 3
  34. 2025Vision-based 3D Semantic Scene Completion via Capture Dynamic Representationscited by 2
  35. 2025Global-Aware Monocular Semantic Scene Completion with State Space Modelscited by 2
  36. 2025Point-based Instance Completion with Scene Constraintscited by 1
  37. 2023VioLA: Aligning Videos to 2D LiDAR Scanscited by 0
  38. 2024Towards Balanced RGB-TSDF Fusion for Consistent Semantic Scene Completion by 3D RGB Feature Completion and a Classwise Entropy Loss Functioncited by 0

50 most cited of 54 All 54 (JSON)