mlsanityData to 7 Oct 2026

Panoptic 3D Scene Reconstruction From a Single RGB Image

arXiv 2111.02444 ↗First posted 3 Nov 2021cs.CVcited by 70

Not on the map. Why?

Abstract (arXiv)

Understanding 3D scenes from a single image is fundamental to a wide variety of tasks, such as for robotics, motion planning, or augmented reality. Existing works in 3D perception from a single RGB image tend to focus on geometric reconstruction only, or geometric reconstruction with semantic segmentation or instance segmentation. Inspired by 2D panoptic segmentation, we propose to unify the tasks of geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation into the task of panoptic 3D scene reconstruction - from a single RGB image, predicting the complete geometric reconstruction of the scene in the camera frustum of the image, along with semantic and instance segmentations. We thus propose a new approach for holistic 3D scene understanding from a single RGB image which learns to lift and propagate 2D features from an input image to a 3D volumetric scene representation. We demonstrate that this holistic view of joint scene reconstruction, semantic, and instance segmentation is beneficial over treating the tasks independently, thus outperforming alternative approaches.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 37 of its 42 references that are in the map. First posted on arXiv 3 Nov 2021.

Builds on

37 of 42 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2017Mask R-CNNcited by 7,259
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  6. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  7. 2018Panoptic Segmentationcited by 796
  8. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  9. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  10. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  11. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  12. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
25 more
  1. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  2. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 306
  3. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 216
  4. 2020OccuSeg: Occupancy-aware 3D Instance Segmentationcited by 177
  5. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 170
  6. 2017SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentationcited by 168
  7. 2019Learning Object Bounding Boxes for 3D Instance Segmentation on Point Cloudscited by 156
  8. 2018GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloudcited by 145
  9. 2018Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundariescited by 144
  10. 20203D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentationcited by 140
  11. 20203D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Priorcited by 124
  12. 20183DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentationcited by 118
  13. 2017ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scanscited by 103
  14. 2019PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Thingscited by 97
  15. 20193D Instance Segmentation via Multi-Task Metric Learningcited by 79
  16. 2019SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scanscited by 77
  17. 2019BlenderProccited by 76
  18. 2018Holistic 3D Scene Parsing and Reconstruction from a Single RGB Imagecited by 69
  19. 2020Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrievecited by 65
  20. 2018Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimationcited by 51
  21. 2020CoReNet: Coherent 3D scene reconstruction from a single RGB imagecited by 45
  22. 2020RfD-Net: Point Scene Understanding by Semantic Instance Reconstructioncited by 36
  23. 2019RevealNet: Seeing Behind Objects in RGB-D Scanscited by 30
  24. 2019Joint Embedding of 3D Scan and CAD Objectscited by 19
  25. 2019PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Pointscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 12; 2023: 15; 2024: 13; 2025: 13; 2026: 14.

70 papers.

Sort:

By area:

  1. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 357
  2. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  3. 2022Panoptic Lifting for 3D Scene Understanding with Neural Fieldscited by 171
  4. 2024MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generationcited by 113
  5. 2025CAST: Component-Aligned 3D Scene Reconstruction from an RGB Imagecited by 101
  6. 2023NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Spacecited by 88
  7. 2023OccDepth: A Depth-Aware Method for 3D Semantic Scene Completioncited by 85
  8. 2025PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformerscited by 78
  9. 2022AutoRF: Learning 3D Object Radiance Fields from Single View Observationscited by 59
  10. 2024Gen3DSR: Generalizable 3D Scene Reconstruction via Divide and Conquer from a Single Viewcited by 59
  11. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  12. 2023NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detectioncited by 51
38 more
  1. 2022SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fieldscited by 49
  2. 2023PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awarenesscited by 42
  3. 2024Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assemblycited by 34
  4. 2023Mesh2Tex: Generating Mesh Textures from Image Queriescited by 31
  5. 2024Monocular Occupancy Prediction for Scalable Indoor Scenescited by 29
  6. 2025Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splattingcited by 29
  7. 2022Learning 3D Scene Priors with 2D Supervisioncited by 24
  8. 2025DepR: Depth Guided Single-view Scene Reconstruction with Instance-level Diffusioncited by 23
  9. 2021Pose2Room: Understanding 3D Scenes from Human Activitiescited by 22
  10. 2023BUOL: A Bottom-Up Framework with Occupancy-aware Lifting for Panoptic 3D Scene Reconstruction From A Single Imagecited by 20
  11. 2025PanSt3R: Multi-view Consistent Panoptic Segmentationcited by 19
  12. 2026ShapeR: Robust Conditional 3D Shape Generation from Casual Capturescited by 19
  13. 2023Panoptic Vision-Language Feature Fieldscited by 17
  14. 2024PLGS: Robust Panoptic Lifting with 3D Gaussian Splattingcited by 17
  15. 2025PanoGS: Gaussian-based Panoptic Segmentation for 3D Open Vocabulary Scene Understandingcited by 16
  16. 2023Mask3D: Pre-training 2D Vision Transformers by Learning Masked 3D Priorscited by 15
  17. 2024PCF-Lift: Panoptic Lifting by Probabilistic Contrastive Fusioncited by 15
  18. 2025SceneMaker: Open-set 3D Scene Generation with Decoupled De-occlusion and Pose Estimation Modelcited by 15
  19. 2023PCR-CG: Point Cloud Registration via Deep Explicit Color and Geometrycited by 13
  20. 20253D-RE-GEN: 3D Reconstruction of Indoor Scenes with a Generative Frameworkcited by 10
  21. 2024PanoSSC: Exploring Monocular Panoptic 3D Scene Reconstruction for Autonomous Drivingcited by 9
  22. 20263D-Fixer: Coarse-to-Fine In-place Completion for 3D Scenes from a Single Imagecited by 9
  23. 2024Coherent 3D Scene Diffusion From a Single RGB Imagecited by 8
  24. 2022Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objectscited by 7
  25. 20243DFIRES: Few Image 3D REconstruction for Scenes with Hidden Surfacecited by 6
  26. 2022Safety Verification for Neural Networks Based on Set-boundary Analysiscited by 5
  27. 2024Semantic Is Enough: Only Semantic Information For NeRF Reconstructioncited by 5
  28. 2025ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editingcited by 5
  29. 2025CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoningcited by 5
  30. 2023Repairing Deep Neural Networks Based on Behavior Imitationcited by 4
  31. 2024UniPlane: Unified Plane Detection and Reconstruction from Posed Monocular Videoscited by 3
  32. 2026PixARMesh: Autoregressive Mesh-Native Single-View Scene Reconstructioncited by 3
  33. 2026AdaSFormer: Adaptive Serialized Transformers for Monocular Semantic Scene Completion from Indoor Environmentscited by 3
  34. 2023Physically Plausible 3D Human-Scene Reconstruction from Monocular RGB Image using an Adversarial Learning Approachcited by 2
  35. 2024Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospectscited by 2
  36. 2026Clean-GS: Semantic Mask-Guided Pruning for 3D Gaussian Splattingcited by 2
  37. 2021Joint stereo 3D object detection and implicit surface reconstructioncited by 1
  38. 2022Neural Adaptive SCEne Tracingcited by 1

50 most cited of 70 All 70 (JSON)