# DepthGAN: GAN-based Depth Generation of Indoor Scenes from Semantic Layouts

arXiv: https://arxiv.org/abs/2203.11453 · HTML: https://mlsanity.com/paper/2203.11453/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [Geometric GAN](https://mlsanity.com/paper/1705.02894/) (2017)
- [Focal Frequency Loss for Image Reconstruction and Synthesis](https://mlsanity.com/paper/2012.12821/) (2020)
- [StyleSwin: Transformer-based GAN for High-resolution Image Generation](https://mlsanity.com/paper/2112.10762/) (2021)
- [ViTGAN: Training GANs with Vision Transformers](https://mlsanity.com/paper/2107.04589/) (2021)
- [Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis](https://mlsanity.com/paper/1910.06809/) (2019)
- [The Edge of Depth: Explicit Constraints between Segmentation and Depth](https://mlsanity.com/paper/2004.00171/) (2020)
- [Improved Transformer for High-Resolution GANs](https://mlsanity.com/paper/2106.07631/) (2021)
- [Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification](https://mlsanity.com/paper/2004.01301/) (2020)
- [End-to-End Optimization of Scene Layout](https://mlsanity.com/paper/2007.11744/) (2020)
- [Unsupervised Learning of Fine Structure Generation for 3D Point Clouds by 2D Projection Matching](https://mlsanity.com/paper/2108.03746/) (2021)

## Built on by (most cited first, 2 of 2)

- [BallGAN: 3D-aware Image Synthesis with a Spherical Background](https://mlsanity.com/paper/2301.09091/) (2023)
- [ContraNeRF: 3D-Aware Generative Model via Contrastive Learning with Unsupervised Implicit Pose Embedding](https://mlsanity.com/paper/2304.14005/) (2023)

## Abstract (arXiv)

Limited by the computational efficiency and accuracy, generating complex 3D scenes remains a challenging problem for existing generation networks. In this work, we propose DepthGAN, a novel method of generating depth maps with only semantic layouts as input. First, we introduce a well-designed cascade of transformer blocks as our generator to capture the structural correlations in depth maps, which makes a balance between global feature aggregation and local attention. Meanwhile, we propose a cross-attention fusion module to guide edge preservation efficiently in depth generation, which exploits additional appearance supervision information. Finally, we conduct extensive experiments on the perspective views of the Structured3d panorama dataset and demonstrate that our DepthGAN achieves superior performance both on quantitative results and visual effects in the depth generation task.Furthermore, 3D indoor scenes can be reconstructed by our generated depth maps with reasonable structure and spatial coherency.