# Learning 3D Scene Priors with 2D Supervision

arXiv: https://arxiv.org/abs/2211.14157 · HTML: https://mlsanity.com/paper/2211.14157/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 63 references that are in the map.
- First posted on arXiv 25 Nov 2022.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics](https://mlsanity.com/paper/2011.09127/) (2020)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [ATISS: Autoregressive Transformers for Indoor Scene Synthesis](https://mlsanity.com/paper/2110.03675/) (2021)
- [Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image](https://mlsanity.com/paper/2002.12212/) (2020)
- [SceneFormer: Indoor Scene Generation with Transformers](https://mlsanity.com/paper/2012.09793/) (2020)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [End-to-end people detection in crowded scenes](https://mlsanity.com/paper/1506.04878/) (2015)
- [Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models](https://mlsanity.com/paper/1811.12463/) (2018)
- [Holistic 3D Scene Understanding from a Single Image with Implicit Representation](https://mlsanity.com/paper/2103.06422/) (2021)
- [IM2CAD](https://mlsanity.com/paper/1608.05137/) (2016)
- [Human-centric Indoor Scene Synthesis Using Stochastic Grammar](https://mlsanity.com/paper/1808.08473/) (2018)
- [Panoptic 3D Scene Reconstruction From a Single RGB Image](https://mlsanity.com/paper/2111.02444/) (2021)
- [Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image](https://mlsanity.com/paper/1808.02201/) (2018)
- [ROCA: Robust CAD Model Retrieval and Alignment from a Single Image](https://mlsanity.com/paper/2112.01988/) (2021)
- [Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve](https://mlsanity.com/paper/2007.13034/) (2020)
- [Deep Generative Modeling for Scene Synthesis via Hybrid Representations](https://mlsanity.com/paper/1808.02084/) (2018)
- [Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene](https://mlsanity.com/paper/1712.01812/) (2017)
- [CoReNet: Coherent 3D scene reconstruction from a single RGB image](https://mlsanity.com/paper/2004.12989/) (2020)
- [Towards High-Fidelity Single-view Holistic Reconstruction of Indoor Scenes](https://mlsanity.com/paper/2207.08656/) (2022)
- [Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image](https://mlsanity.com/paper/2108.09368/) (2021)
- [GRAINS: Generative Recursive Autoencoders for INdoor Scenes](https://mlsanity.com/paper/1807.09193/) (2018)
- [Adversarial Parametric Pose Prior](https://mlsanity.com/paper/2112.04203/) (2021)
- [RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction](https://mlsanity.com/paper/2011.14744/) (2020)

## Built on by (most cited first, 24 of 24)

- [DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis](https://mlsanity.com/paper/2303.14207/) (2023)
- [PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI](https://mlsanity.com/paper/2404.09465/) (2024)
- [CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP](https://mlsanity.com/paper/2303.04748/) (2023)
- [CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion](https://mlsanity.com/paper/2305.16283/) (2023)
- [Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assembly](https://mlsanity.com/paper/2410.15971/) (2024)
- [NAP: Neural 3D Articulation Prior](https://mlsanity.com/paper/2305.16315/) (2023)
- [SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation](https://mlsanity.com/paper/2412.01801/) (2024)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [RICO: Regularizing the Unobservable for Indoor Compositional Reconstruction](https://mlsanity.com/paper/2303.08605/) (2023)
- [DeBaRA: Denoising-Based 3D Room Arrangement Generation](https://mlsanity.com/paper/2409.18336/) (2024)
- [RoomDesigner: Encoding Anchor-latents for Style-consistent and Shape-compatible Indoor Scene Generation](https://mlsanity.com/paper/2310.10027/) (2023)
- [FurniScene: A Large-scale 3D Room Dataset with Intricate Furnishing Scenes](https://mlsanity.com/paper/2401.03470/) (2024)
- [External Knowledge Enhanced 3D Scene Generation from Sketch](https://mlsanity.com/paper/2403.14121/) (2024)
- [HouseCrafter: Lifting Floorplans to 3D Scenes with 2D Diffusion Model](https://mlsanity.com/paper/2406.20077/) (2024)
- [Monocular Human-Object Reconstruction in the Wild](https://mlsanity.com/paper/2407.20566/) (2024)
- [Unsupervised Template-assisted Point Cloud Shape Correspondence Network](https://mlsanity.com/paper/2403.16412/) (2024)
- [S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Field](https://mlsanity.com/paper/2412.17561/) (2024)
- [Physically Plausible 3D Human-Scene Reconstruction from Monocular RGB Image using an Adversarial Learning Approach](https://mlsanity.com/paper/2307.14570/) (2023)
- [3D Scene Diffusion Guidance using Scene Graphs](https://mlsanity.com/paper/2308.04468/) (2023)
- [2D-3D Attention and Entropy for Pose Robust 2D Facial Recognition](https://mlsanity.com/paper/2505.09073/) (2025)
- [FlowObject: Flow Steering for Bridging Generative Priors and Reconstruction Fidelity](https://mlsanity.com/paper/2606.19019/) (2026)
- [HierRelTriple: Guiding Indoor Layout Generation with Hierarchical Relationship Triplet Losses](https://mlsanity.com/paper/2503.20289/) (2025)
- [LPA3D: 3D Room-Level Scene Generation from In-the-Wild Images](https://mlsanity.com/paper/2504.02337/) (2025)
- [GLINT: Modeling Scene-Scale Transparency via Gaussian Radiance Transport](https://mlsanity.com/paper/2603.26181/) (2026)

## Abstract (arXiv)

Holistic 3D scene understanding entails estimation of both layout configuration and object geometry in a 3D environment. Recent works have shown advances in 3D scene estimation from various input modalities (e.g., images, 3D scans), by leveraging 3D supervision (e.g., 3D bounding boxes or CAD models), for which collection at scale is expensive and often intractable. To address this shortcoming, we propose a new method to learn 3D scene priors of layout and shape without requiring any 3D ground truth. Instead, we rely on 2D supervision from multi-view RGB images. Our method represents a 3D scene as a latent vector, from which we can progressively decode to a sequence of objects characterized by their class categories, 3D bounding boxes, and meshes. With our trained autoregressive decoder representing the scene prior, our method facilitates many downstream applications, including scene synthesis, interpolation, and single-view reconstruction. Experiments on 3D-FRONT and ScanNet show that our method outperforms state of the art in single-view reconstruction, and achieves state-of-the-art results in scene synthesis against baselines which require for 3D supervision.