mlsanityData to 14 Oct 2026

Learning 3D Scene Priors with 2D Supervision

arXiv 2211.14157 ↗First posted 25 Nov 2022cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

Holistic 3D scene understanding entails estimation of both layout configuration and object geometry in a 3D environment. Recent works have shown advances in 3D scene estimation from various input modalities (e.g., images, 3D scans), by leveraging 3D supervision (e.g., 3D bounding boxes or CAD models), for which collection at scale is expensive and often intractable. To address this shortcoming, we propose a new method to learn 3D scene priors of layout and shape without requiring any 3D ground truth. Instead, we rely on 2D supervision from multi-view RGB images. Our method represents a 3D scene as a latent vector, from which we can progressively decode to a sequence of objects characterized by their class categories, 3D bounding boxes, and meshes. With our trained autoregressive decoder representing the scene prior, our method facilitates many downstream applications, including scene synthesis, interpolation, and single-view reconstruction. Experiments on 3D-FRONT and ScanNet show that our method outperforms state of the art in single-view reconstruction, and achieves state-of-the-art results in scene synthesis against baselines which require for 3D supervision.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 38 of its 63 references that are in the map. First posted on arXiv 25 Nov 2022.

Builds on

38 of 63 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020End-to-End Object Detection with Transformerscited by 7,414
  3. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  4. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  6. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 784
  7. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  8. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  9. 2018Pix3D: Dataset and Methods for Single-Image 3D Shape Modelingcited by 218
  10. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 200
  11. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 172
  12. 2020SceneFormer: Indoor Scene Generation with Transformerscited by 158
26 more
  1. 2018Scan2CAD: Learning CAD Model Alignment in RGB-D Scanscited by 142
  2. 2015End-to-end people detection in crowded scenescited by 140
  3. 2018Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Modelscited by 106
  4. 2021Holistic 3D Scene Understanding from a Single Image with Implicit Representationcited by 94
  5. 2016IM2CADcited by 78
  6. 2018Human-centric Indoor Scene Synthesis Using Stochastic Grammarcited by 74
  7. 2021Panoptic 3D Scene Reconstruction From a Single RGB Imagecited by 70
  8. 2018Holistic 3D Scene Parsing and Reconstruction from a Single RGB Imagecited by 69
  9. 2021ROCA: Robust CAD Model Retrieval and Alignment from a Single Imagecited by 67
  10. 2020Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrievecited by 65
  11. 2018Deep Generative Modeling for Scene Synthesis via Hybrid Representationscited by 63
  12. 2017Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scenecited by 50
  13. 2020CoReNet: Coherent 3D scene reconstruction from a single RGB imagecited by 45
  14. 2022Towards High-Fidelity Single-view Holistic Reconstruction of Indoor Scenescited by 44
  15. 2021Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Imagecited by 39
  16. 2018GRAINS: Generative Recursive Autoencoders for INdoor Scenescited by 38
  17. 2021Adversarial Parametric Pose Priorcited by 38
  18. 2020RfD-Net: Point Scene Understanding by Semantic Instance Reconstructioncited by 36
  19. 2019RevealNet: Seeing Behind Objects in RGB-D Scanscited by 30
  20. 2021Scene Synthesis via Uncertainty-Driven Attribute Synchronizationcited by 29
  21. 2021Indoor Scene Generation from a Collection of Semantic-Segmented Depth Imagescited by 27
  22. 2021Pose2Room: Understanding 3D Scenes from Human Activitiescited by 22
  23. 2017SceneSeer: 3D Scene Design with Natural Languagecited by 21
  24. 2022Learning 3D Object Shape and Layout without 3D Supervisioncited by 19
  25. 2020SceneGen: Generative Contextual Scene Augmentation using Scene Graph Priorscited by 11
  26. 2017SceneSuggest: Context-driven 3D Scene Designcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 10; 2025: 4; 2026: 2.

24 papers.

Sort:

By area:

  1. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 166
  2. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 125
  3. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  4. 2023CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusioncited by 54
  5. 2024Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assemblycited by 35
  6. 2023NAP: Neural 3D Articulation Priorcited by 25
  7. 2024SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generationcited by 24
  8. 20253D Scene Generation: A Surveycited by 24
  9. 2023RICO: Regularizing the Unobservable for Indoor Compositional Reconstructioncited by 20
  10. 2024DeBaRA: Denoising-Based 3D Room Arrangement Generationcited by 20
  11. 2023RoomDesigner: Encoding Anchor-latents for Style-consistent and Shape-compatible Indoor Scene Generationcited by 12
  12. 2024FurniScene: A Large-scale 3D Room Dataset with Intricate Furnishing Scenescited by 10
12 more
  1. 2024External Knowledge Enhanced 3D Scene Generation from Sketchcited by 9
  2. 2024HouseCrafter: Lifting Floorplans to 3D Scenes with 2D Diffusion Modelcited by 9
  3. 2024Monocular Human-Object Reconstruction in the Wildcited by 9
  4. 2024Unsupervised Template-assisted Point Cloud Shape Correspondence Networkcited by 7
  5. 2024S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Fieldcited by 3
  6. 2023Physically Plausible 3D Human-Scene Reconstruction from Monocular RGB Image using an Adversarial Learning Approachcited by 2
  7. 20233D Scene Diffusion Guidance using Scene Graphscited by 1
  8. 20252D-3D Attention and Entropy for Pose Robust 2D Facial Recognitioncited by 1
  9. 2026FlowObject: Flow Steering for Bridging Generative Priors and Reconstruction Fidelitycited by 1
  10. 2025HierRelTriple: Guiding Indoor Layout Generation with Hierarchical Relationship Triplet Lossescited by 0
  11. 2025LPA3D: 3D Room-Level Scene Generation from In-the-Wild Imagescited by 0
  12. 2026GLINT: Modeling Scene-Scale Transparency via Gaussian Radiance Transportcited by 0