# 3D-Aware Indoor Scene Synthesis with Depth Priors

arXiv: https://arxiv.org/abs/2202.08553 · HTML: https://mlsanity.com/paper/2202.08553/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 51 references that are in the map.
- First posted on arXiv 17 Feb 2022.

## Builds on (most cited first, 30 of 35)

- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [Closed-Form Factorization of Latent Semantics in GANs](https://mlsanity.com/paper/2007.06600/) (2020)
- [InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs](https://mlsanity.com/paper/2005.09635/) (2020)
- [StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation](https://mlsanity.com/paper/2112.11427/) (2021)
- [Learning to Recover 3D Scene Shape from a Single Image](https://mlsanity.com/paper/2012.09365/) (2020)
- [GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation](https://mlsanity.com/paper/2112.08867/) (2021)
- [BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images](https://mlsanity.com/paper/2002.08988/) (2020)
- [Unconstrained Scene Generation with Locally Conditioned Radiance Fields](https://mlsanity.com/paper/2104.00670/) (2021)
- [Visual Object Networks: Image Generation with Disentangled 3D Representation](https://mlsanity.com/paper/1812.02725/) (2018)
- [3D-aware Image Synthesis via Learning Structural and Textural Representations](https://mlsanity.com/paper/2112.10759/) (2021)
- [Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis](https://mlsanity.com/paper/1911.09267/) (2019)

## Built on by (most cited first, 30 of 32)

- [Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion](https://mlsanity.com/paper/2212.06135/) (2022)
- [CityDreamer: Compositional Generative Model of Unbounded 3D Cities](https://mlsanity.com/paper/2309.00610/) (2023)
- [SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections](https://mlsanity.com/paper/2302.01330/) (2023)
- [3D-aware Image Generation using 2D Diffusion Models](https://mlsanity.com/paper/2303.17905/) (2023)
- [DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis](https://mlsanity.com/paper/2212.11984/) (2022)
- [3D generation on ImageNet](https://mlsanity.com/paper/2303.01416/) (2023)
- [StyleGAN knows Normal, Depth, Albedo, and More](https://mlsanity.com/paper/2306.00987/) (2023)
- [Generative Gaussian Splatting for Unbounded 3D City Generation](https://mlsanity.com/paper/2406.06526/) (2024)
- [A General Protocol to Probe Large Vision Models for 3D Physical Understanding](https://mlsanity.com/paper/2310.06836/) (2023)
- [Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answering](https://mlsanity.com/paper/2209.10326/) (2022)
- [SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation](https://mlsanity.com/paper/2412.01801/) (2024)
- [CityGen: Infinite and Controllable City Layout Generation](https://mlsanity.com/paper/2312.01508/) (2023)
- [WorldGen: From Text to Traversable and Interactive 3D Worlds](https://mlsanity.com/paper/2511.16825/) (2025)
- [A Survey on Deep Generative 3D-aware Image Synthesis](https://mlsanity.com/paper/2210.14267/) (2022)
- [Tri2-plane: Thinking Head Avatar via Feature Pyramid](https://mlsanity.com/paper/2401.09386/) (2024)
- [LinkGAN: Linking GAN Latents to Pixels for Controllable Image Synthesis](https://mlsanity.com/paper/2301.04604/) (2023)
- [Denoising Diffusion via Image-Based Rendering](https://mlsanity.com/paper/2402.03445/) (2024)
- [Learning 3D-aware Image Synthesis with Unknown Pose Distribution](https://mlsanity.com/paper/2301.07702/) (2023)
- [Efficient 3D Articulated Human Generation with Layered Surface Volumes](https://mlsanity.com/paper/2307.05462/) (2023)
- [DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2212.08861/) (2022)
- [SinGRAF: Learning a 3D Generative Radiance Field for a Single Scene](https://mlsanity.com/paper/2211.17260/) (2022)
- [BlobGAN-3D: A Spatially-Disentangled 3D-Aware Generative Model for Indoor Scenes](https://mlsanity.com/paper/2303.14706/) (2023)
- [From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries](https://mlsanity.com/paper/2510.10292/) (2025)
- [G3DR: Generative 3D Reconstruction in ImageNet](https://mlsanity.com/paper/2403.00939/) (2024)
- [GLeaD: Improving GANs with A Generator-Leading Task](https://mlsanity.com/paper/2212.03752/) (2022)
- [Benchmarking and Analyzing 3D-aware Image Synthesis with a Modularized Codebase](https://mlsanity.com/paper/2306.12423/) (2023)
- [BerfScene: Bev-conditioned Equivariant Radiance Fields for Infinite 3D Scene Generation](https://mlsanity.com/paper/2312.02136/) (2023)
- [Consistent Multimodal Generation via A Unified GAN Framework](https://mlsanity.com/paper/2307.01425/) (2023)
- [SideGAN: 3D-Aware Generative Model for Improved Side-View Image Synthesis](https://mlsanity.com/paper/2309.10388/) (2023)
- [Deep Generative Models on 3D Representations: A Survey](https://mlsanity.com/paper/2210.15663/) (2022)

## Abstract (arXiv)

Despite the recent advancement of Generative Adversarial Networks (GANs) in learning 3D-aware image synthesis from 2D data, existing methods fail to model indoor scenes due to the large diversity of room layouts and the objects inside. We argue that indoor scenes do not have a shared intrinsic structure, and hence only using 2D images cannot adequately guide the model with the 3D geometry. In this work, we fill in this gap by introducing depth as a 3D prior. Compared with other 3D data formats, depth better fits the convolution-based generation mechanism and is more easily accessible in practice. Specifically, we propose a dual-path generator, where one path is responsible for depth generation, whose intermediate features are injected into the other path as the condition for appearance rendering. Such a design eases the 3D-aware synthesis with explicit geometry information. Meanwhile, we introduce a switchable discriminator both to differentiate real v.s. fake domains and to predict the depth from a given input. In this way, the discriminator can take the spatial arrangement into account and advise the generator to learn an appropriate depth condition. Extensive experimental results suggest that our approach is capable of synthesizing indoor scenes with impressively good quality and 3D consistency, significantly outperforming state-of-the-art alternatives.