# CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion

arXiv: https://arxiv.org/abs/2305.16283 · HTML: https://mlsanity.com/paper/2305.16283/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 59 of its 76 references that are in the map.
- First posted on arXiv 25 May 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 59)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis](https://mlsanity.com/paper/2212.05032/) (2022)
- [3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics](https://mlsanity.com/paper/2011.09127/) (2020)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [3D-FUTURE: 3D Furniture shape with TextURE](https://mlsanity.com/paper/2009.09633/) (2020)
- [SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation](https://mlsanity.com/paper/2212.04493/) (2022)
- [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019)
- [Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions](https://mlsanity.com/paper/2004.03967/) (2020)
- [ATISS: Autoregressive Transformers for Indoor Scene Synthesis](https://mlsanity.com/paper/2110.03675/) (2021)
- [RIO: 3D Object Instance Re-Localization in Changing Indoor Environments](https://mlsanity.com/paper/1908.06109/) (2019)
- [Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization](https://mlsanity.com/paper/2201.13360/) (2022)

## Built on by (most cited first, 30 of 54)

- [SceneGen: Single-Image 3D Scene Generation in One Feedforward Pass](https://mlsanity.com/paper/2508.15769/) (2025)
- [SAGE: Scalable Agentic 3D Scene Generation for Embodied AI](https://mlsanity.com/paper/2602.10116/) (2026)
- [Global-Local Tree Search in VLMs for 3D Indoor Scene Generation](https://mlsanity.com/paper/2503.18476/) (2025)
- [SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesis](https://mlsanity.com/paper/2503.14756/) (2025)
- [SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation](https://mlsanity.com/paper/2412.01801/) (2024)
- [HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generation](https://mlsanity.com/paper/2503.16848/) (2025)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [WorldCraft: Photo-Realistic 3D World Creation and Customization via LLM Agents](https://mlsanity.com/paper/2502.15601/) (2025)
- [X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability](https://mlsanity.com/paper/2506.13558/) (2025)
- [HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation](https://mlsanity.com/paper/2504.13072/) (2025)
- [SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning](https://mlsanity.com/paper/2605.09423/) (2026)
- [Agentic 3D Scene Generation with Spatially Contextualized VLMs](https://mlsanity.com/paper/2505.20129/) (2025)
- [Imaginarium: Vision-guided High-Quality 3D Scene Layout Generation](https://mlsanity.com/paper/2510.15564/) (2025)
- [FurniScene: A Large-scale 3D Room Dataset with Intricate Furnishing Scenes](https://mlsanity.com/paper/2401.03470/) (2024)
- [CHOrD: Generation of Collision-Free, House-Scale, and Organized Digital Twins for 3D Indoor Scenes with Controllable Floor Plans and Optimal Layouts](https://mlsanity.com/paper/2503.11958/) (2025)
- [FROSS: Faster-than-Real-Time Online 3D Semantic Scene Graph Generation from RGB-D Images](https://mlsanity.com/paper/2507.19993/) (2025)
- [La La LiDAR: Large-Scale Layout Generation from LiDAR Data](https://mlsanity.com/paper/2508.03691/) (2025)
- [Controllable 3D Outdoor Scene Generation via Scene Graphs](https://mlsanity.com/paper/2503.07152/) (2025)
- [ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editing](https://mlsanity.com/paper/2506.02459/) (2025)
- [Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimization](https://mlsanity.com/paper/2507.14841/) (2025)
- [SemLayoutDiff: Semantic Layout Generation with Diffusion Model for Indoor Scene Synthesis](https://mlsanity.com/paper/2508.18597/) (2025)
- [M3DLayout: A Multi-Source Dataset of 3D Indoor Layouts and Structured Descriptions for 3D Generation](https://mlsanity.com/paper/2509.23728/) (2025)
- [Recent Advances in 3D Object and Scene Generation: A Survey](https://mlsanity.com/paper/2504.11734/) (2025)
- [Foundation Visual Encoders Are Secretly Few-Shot Anomaly Detectors](https://mlsanity.com/paper/2510.01934/) (2025)
- [3D Software Synthesis Driven by Constraint-Expressive Intermediate Representation](https://mlsanity.com/paper/2507.18625/) (2025)
- [SyncSDE: A Probabilistic Framework for Diffusion Synchronization](https://mlsanity.com/paper/2503.21555/) (2025)
- [Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editing](https://mlsanity.com/paper/2603.17583/) (2026)
- [Diffusion-based Graph Generative Methods](https://mlsanity.com/paper/2401.15617/) (2024)
- [SceneNAT: Masked Generative Modeling for Language-Guided Indoor Scene Synthesis](https://mlsanity.com/paper/2601.07218/) (2026)
- [SceneConductor: 3D Scene Generation from a Single Image with Multi-Agent Orchestration](https://mlsanity.com/paper/2606.08402/) (2026)

## Abstract (arXiv)

Controllable scene synthesis aims to create interactive environments for various industrial use cases. Scene graphs provide a highly suitable interface to facilitate these applications by abstracting the scene context in a compact manner. Existing methods, reliant on retrieval from extensive databases or pre-trained shape embeddings, often overlook scene-object and object-object relationships, leading to inconsistent results due to their limited generation capacity. To address this issue, we present CommonScenes, a fully generative model that converts scene graphs into corresponding controllable 3D scenes, which are semantically realistic and conform to commonsense. Our pipeline consists of two branches, one predicting the overall scene layout via a variational auto-encoder and the other generating compatible shapes via latent diffusion, capturing global scene-object and local inter-object relationships in the scene graph while preserving shape diversity. The generated scenes can be manipulated by editing the input scene graph and sampling the noise in the diffusion model. Due to lacking a scene graph dataset offering high-quality object-level meshes with relations, we also construct SG-FRONT, enriching the off-the-shelf indoor dataset 3D-FRONT with additional scene graph labels. Extensive experiments are conducted on SG-FRONT where CommonScenes shows clear advantages over other methods regarding generation consistency, quality, and diversity. Codes and the dataset will be released upon acceptance.