# End-to-End Optimization of Scene Layout

arXiv: https://arxiv.org/abs/2007.11744 · HTML: https://mlsanity.com/paper/2007.11744/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2020.

## Built on by (most cited first, 30 of 52)

- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [ATISS: Autoregressive Transformers for Indoor Scene Synthesis](https://mlsanity.com/paper/2110.03675/) (2021)
- [InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior](https://mlsanity.com/paper/2402.04717/) (2024)
- [Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs](https://mlsanity.com/paper/2108.08841/) (2021)
- [LEGO-Net: Learning Regular Rearrangements of Objects in Rooms](https://mlsanity.com/paper/2301.09629/) (2023)
- [DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents](https://mlsanity.com/paper/2101.11796/) (2021)
- [EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusion](https://mlsanity.com/paper/2405.00915/) (2024)
- [Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation](https://mlsanity.com/paper/2505.02836/) (2025)
- [MIME: Human-Aware 3D Scene Generation](https://mlsanity.com/paper/2212.04360/) (2022)
- [CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion](https://mlsanity.com/paper/2305.16283/) (2023)
- [RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models](https://mlsanity.com/paper/2212.05993/) (2022)
- [SceneHGN: Hierarchical Graph Networks for 3D Indoor Scene Generation with Fine-Grained Geometry](https://mlsanity.com/paper/2302.10237/) (2023)
- [LLplace: The 3D Indoor Scene Layout Generation and Editing via Large Language Model](https://mlsanity.com/paper/2406.03866/) (2024)
- [Indoor Scene Generation from a Collection of Semantic-Segmented Depth Images](https://mlsanity.com/paper/2108.09022/) (2021)
- [SurfGen: Adversarial 3D Shape Synthesis with Explicit Surface Discriminators](https://mlsanity.com/paper/2201.00112/) (2022)
- [3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs](https://mlsanity.com/paper/2209.07896/) (2022)
- [SG-Bot: Object Rearrangement via Coarse-to-Fine Robotic Imagination on Scene Graphs](https://mlsanity.com/paper/2309.12188/) (2023)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [Chat2Layout: Interactive 3D Furniture Layout with a Multimodal LLM](https://mlsanity.com/paper/2407.21333/) (2024)
- [LayoutEnhancer: Generating Good Indoor Layouts from Imperfect Data](https://mlsanity.com/paper/2202.00185/) (2022)
- [Scene Synthesis from Human Motion](https://mlsanity.com/paper/2301.01424/) (2023)
- [Graph Transformer GANs for Graph-Constrained House Generation](https://mlsanity.com/paper/2303.08225/) (2023)
- [CLIP-Layout: Style-Consistent Indoor Scene Synthesis with Semantic Furniture Embedding](https://mlsanity.com/paper/2303.03565/) (2023)
- [PreGenie: An Agentic Framework for High-quality Visual Presentation Generation](https://mlsanity.com/paper/2505.21660/) (2025)
- [LUMINOUS: Indoor Scene Generation for Embodied AI Challenges](https://mlsanity.com/paper/2111.05527/) (2021)
- [Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation](https://mlsanity.com/paper/2410.00447/) (2024)
- [FurniScene: A Large-scale 3D Room Dataset with Intricate Furnishing Scenes](https://mlsanity.com/paper/2401.03470/) (2024)
- [Exploiting Relationship for Complex-scene Image Generation](https://mlsanity.com/paper/2104.00356/) (2021)
- [Spatial Commonsense Graph for Object Localisation in Partial Scenes](https://mlsanity.com/paper/2203.05380/) (2022)
- [Planner3D: LLM-enhanced graph prior meets 3D indoor scene explicit regularization](https://mlsanity.com/paper/2403.12848/) (2024)

## Abstract (arXiv)

We propose an end-to-end variational generative model for scene layout synthesis conditioned on scene graphs. Unlike unconditional scene layout generation, we use scene graphs as an abstract but general representation to guide the synthesis of diverse scene layouts that satisfy relationships included in the scene graph. This gives rise to more flexible control over the synthesis process, allowing various forms of inputs such as scene layouts extracted from sentences or inferred from a single color image. Using our conditional layout synthesizer, we can generate various layouts that share the same structure of the input example. In addition to this conditional generation design, we also integrate a differentiable rendering module that enables layout refinement using only 2D projections of the scene. Given a depth and a semantics map, the differentiable rendering module enables optimizing over the synthesized layout to fit the given input in an analysis-by-synthesis fashion. Experiments suggest that our model achieves higher accuracy and diversity in conditional scene synthesis and allows exemplar-based scene generation from various input forms.