# Indoor Scene Generation from a Collection of Semantic-Segmented Depth Images

arXiv: https://arxiv.org/abs/2108.09022 · HTML: https://mlsanity.com/paper/2108.09022/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 20 Aug 2021.

## Builds on (most cited first, 20 of 20)

- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency](https://mlsanity.com/paper/1704.06254/) (2017)
- [Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image](https://mlsanity.com/paper/2002.12212/) (2020)
- [3D Shape Induction from 2D Views of Multiple Objects](https://mlsanity.com/paper/1612.05872/) (2016)
- [Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models](https://mlsanity.com/paper/1811.12463/) (2018)
- [SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans](https://mlsanity.com/paper/1912.00036/) (2019)
- [Human-centric Indoor Scene Synthesis Using Stochastic Grammar](https://mlsanity.com/paper/1808.08473/) (2018)
- [Deep Generative Modeling for Scene Synthesis via Hybrid Representations](https://mlsanity.com/paper/1808.02084/) (2018)
- [End-to-End Optimization of Scene Layout](https://mlsanity.com/paper/2007.11744/) (2020)
- [Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations](https://mlsanity.com/paper/1809.04766/) (2018)
- [Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene](https://mlsanity.com/paper/1712.01812/) (2017)
- [CoReNet: Coherent 3D scene reconstruction from a single RGB image](https://mlsanity.com/paper/2004.12989/) (2020)
- [Cascaded Context Pyramid for Full-Resolution 3D Semantic Scene Completion](https://mlsanity.com/paper/1908.00382/) (2019)
- [GRAINS: Generative Recursive Autoencoders for INdoor Scenes](https://mlsanity.com/paper/1807.09193/) (2018)
- [Text to 3D Scene Generation with Rich Lexical Grounding](https://mlsanity.com/paper/1505.06289/) (2015)
- [View-volume Network for Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1806.05361/) (2018)
- [Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View](https://mlsanity.com/paper/1712.04569/) (2017)
- [Synthesizing 3D Shapes from Silhouette Image Collections using Multi-projection Generative Adversarial Networks](https://mlsanity.com/paper/1906.03841/) (2019)

## Built on by (most cited first, 27 of 27)

- [DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis](https://mlsanity.com/paper/2303.14207/) (2023)
- [PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI](https://mlsanity.com/paper/2404.09465/) (2024)
- [InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior](https://mlsanity.com/paper/2402.04717/) (2024)
- [AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homes](https://mlsanity.com/paper/2312.06644/) (2023)
- [CG3D: Compositional Generation for Text-to-3D via Gaussian Splatting](https://mlsanity.com/paper/2311.17907/) (2023)
- [LLplace: The 3D Indoor Scene Layout Generation and Editing via Large Language Model](https://mlsanity.com/paper/2406.03866/) (2024)
- [Mixed Diffusion for 3D Indoor Scene Synthesis](https://mlsanity.com/paper/2405.21066/) (2024)
- [NAP: Neural 3D Articulation Prior](https://mlsanity.com/paper/2305.16315/) (2023)
- [Learning 3D Scene Priors with 2D Supervision](https://mlsanity.com/paper/2211.14157/) (2022)
- [RoomDesigner: Encoding Anchor-latents for Style-consistent and Shape-compatible Indoor Scene Generation](https://mlsanity.com/paper/2310.10027/) (2023)
- [FurniScene: A Large-scale 3D Room Dataset with Intricate Furnishing Scenes](https://mlsanity.com/paper/2401.03470/) (2024)
- [External Knowledge Enhanced 3D Scene Generation from Sketch](https://mlsanity.com/paper/2403.14121/) (2024)
- [FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts](https://mlsanity.com/paper/2506.02781/) (2025)
- [Lay-A-Scene: Personalized 3D Object Arrangement Using Text-to-Image Priors](https://mlsanity.com/paper/2406.00687/) (2024)
- [InstructLayout: Instruction-Driven 2D and 3D Layout Synthesis with Semantic Graph Prior](https://mlsanity.com/paper/2407.07580/) (2024)
- [SemLayoutDiff: Semantic Layout Generation with Diffusion Model for Indoor Scene Synthesis](https://mlsanity.com/paper/2508.18597/) (2025)
- [From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries](https://mlsanity.com/paper/2510.10292/) (2025)
- [ScanEdit: Hierarchically-Guided Functional 3D Scan Editing](https://mlsanity.com/paper/2504.15049/) (2025)
- [3D Software Synthesis Driven by Constraint-Expressive Intermediate Representation](https://mlsanity.com/paper/2507.18625/) (2025)
- [S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Field](https://mlsanity.com/paper/2412.17561/) (2024)
- [Computer-Aided Layout Generation for Building Design: A Review](https://mlsanity.com/paper/2504.09694/) (2025)
- [3D Scene Diffusion Guidance using Scene Graphs](https://mlsanity.com/paper/2308.04468/) (2023)
- [Decorum: A Language-Based Approach For Style-Conditioned Synthesis of Indoor 3D Scenes](https://mlsanity.com/paper/2503.18155/) (2025)
- [GeoSceneGraph: Geometric Scene Graph Diffusion Model for Text-guided 3D Indoor Scene Synthesis](https://mlsanity.com/paper/2511.14884/) (2025)
- [Agentic Designer: Progressive Multi-Agent Collaboration for Structure-Aware Interior Layout Generation](https://mlsanity.com/paper/2607.20866/) (2026)
- [HierRelTriple: Guiding Indoor Layout Generation with Hierarchical Relationship Triplet Losses](https://mlsanity.com/paper/2503.20289/) (2025)
- [LPA3D: 3D Room-Level Scene Generation from In-the-Wild Images](https://mlsanity.com/paper/2504.02337/) (2025)

## Abstract (arXiv)

We present a method for creating 3D indoor scenes with a generative model learned from a collection of semantic-segmented depth images captured from different unknown scenes. Given a room with a specified size, our method automatically generates 3D objects in a room from a randomly sampled latent code. Different from existing methods that represent an indoor scene with the type, location, and other properties of objects in the room and learn the scene layout from a collection of complete 3D indoor scenes, our method models each indoor scene as a 3D semantic scene volume and learns a volumetric generative adversarial network (GAN) from a collection of 2.5D partial observations of 3D scenes. To this end, we apply a differentiable projection layer to project the generated 3D semantic scene volumes into semantic-segmented depth images and design a new multiple-view discriminator for learning the complete 3D scene volume from 2.5D semantic-segmented depth images. Compared to existing methods, our method not only efficiently reduces the workload of modeling and acquiring 3D scenes for training, but also produces better object shapes and their detailed layouts in the scene. We evaluate our method with different indoor scene datasets and demonstrate the advantages of our method. We also extend our method for generating 3D indoor scenes from semantic-segmented depth images inferred from RGB images of real scenes.