# CLIP-Layout: Style-Consistent Indoor Scene Synthesis with Semantic Furniture Embedding

arXiv: https://arxiv.org/abs/2303.03565 · HTML: https://mlsanity.com/paper/2303.03565/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 50 references that are in the map.
- First posted on arXiv 7 Mar 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics](https://mlsanity.com/paper/2011.09127/) (2020)
- [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022)
- [3D-FUTURE: 3D Furniture shape with TextURE](https://mlsanity.com/paper/2009.09633/) (2020)
- [CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation](https://mlsanity.com/paper/2110.02624/) (2021)
- [ATISS: Autoregressive Transformers for Indoor Scene Synthesis](https://mlsanity.com/paper/2110.03675/) (2021)
- [SceneFormer: Indoor Scene Generation with Transformers](https://mlsanity.com/paper/2012.09793/) (2020)
- [GAUDI: A Neural Architect for Immersive 3D Scene Generation](https://mlsanity.com/paper/2207.13751/) (2022)
- [Meta-Sim: Learning to Generate Synthetic Datasets](https://mlsanity.com/paper/1904.11621/) (2019)
- [Fast Transformers with Clustered Attention](https://mlsanity.com/paper/2007.04825/) (2020)
- [Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models](https://mlsanity.com/paper/1811.12463/) (2018)
- [Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data](https://mlsanity.com/paper/1810.10093/) (2018)
- [3DILG: Irregular Latent Grids for 3D Generative Modeling](https://mlsanity.com/paper/2205.13914/) (2022)
- [Human-centric Indoor Scene Synthesis Using Stochastic Grammar](https://mlsanity.com/paper/1808.08473/) (2018)
- [SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections](https://mlsanity.com/paper/2302.01330/) (2023)
- [Deep Generative Modeling for Scene Synthesis via Hybrid Representations](https://mlsanity.com/paper/1808.02084/) (2018)
- [Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation](https://mlsanity.com/paper/2008.09092/) (2020)
- [MIME: Human-Aware 3D Scene Generation](https://mlsanity.com/paper/2212.04360/) (2022)
- [SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentation](https://mlsanity.com/paper/1907.11308/) (2019)
- [End-to-End Optimization of Scene Layout](https://mlsanity.com/paper/2007.11744/) (2020)
- [LayoutEnhancer: Generating Good Indoor Layouts from Imperfect Data](https://mlsanity.com/paper/2202.00185/) (2022)

## Built on by (most cited first, 16 of 16)

- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior](https://mlsanity.com/paper/2402.04717/) (2024)
- [I-Design: Personalized LLM Interior Designer](https://mlsanity.com/paper/2404.02838/) (2024)
- [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024)
- [DeBaRA: Denoising-Based 3D Room Arrangement Generation](https://mlsanity.com/paper/2409.18336/) (2024)
- [Forest2Seq: Revitalizing Order Prior for Sequential Indoor Scene Synthesis](https://mlsanity.com/paper/2407.05388/) (2024)
- [FurniScene: A Large-scale 3D Room Dataset with Intricate Furnishing Scenes](https://mlsanity.com/paper/2401.03470/) (2024)
- [InstructLayout: Instruction-Driven 2D and 3D Layout Synthesis with Semantic Graph Prior](https://mlsanity.com/paper/2407.07580/) (2024)
- [ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editing](https://mlsanity.com/paper/2506.02459/) (2025)
- [Human-Aware 3D Scene Generation with Spatially-constrained Diffusion Models](https://mlsanity.com/paper/2406.18159/) (2024)
- [3D Software Synthesis Driven by Constraint-Expressive Intermediate Representation](https://mlsanity.com/paper/2507.18625/) (2025)
- [SceneNAT: Masked Generative Modeling for Language-Guided Indoor Scene Synthesis](https://mlsanity.com/paper/2601.07218/) (2026)
- [Decorum: A Language-Based Approach For Style-Conditioned Synthesis of Indoor 3D Scenes](https://mlsanity.com/paper/2503.18155/) (2025)
- [BIM-Native Tokenization for Constraint-Aware Room Layout Synthesis](https://mlsanity.com/paper/2512.04832/) (2025)
- [DecoMind: A Generative AI System for Personalized Interior Design Layouts](https://mlsanity.com/paper/2508.16696/) (2025)
- [SPHERE: Adaptive VR Indoor Scene Generation via LLM-Enhanced Spatial Preference Learning and Human-in-the-Loop RL](https://mlsanity.com/paper/2610.02023/) (2026)

## Abstract (arXiv)

Indoor scene synthesis involves automatically picking and placing furniture appropriately on a floor plan, so that the scene looks realistic and is functionally plausible. Such scenes can serve as homes for immersive 3D experiences, or be used to train embodied agents. Existing methods for this task rely on labeled categories of furniture, e.g. bed, chair or table, to generate contextually relevant combinations of furniture. Whether heuristic or learned, these methods ignore instance-level visual attributes of objects, and as a result may produce visually less coherent scenes. In this paper, we introduce an auto-regressive scene model which can output instance-level predictions, using general purpose image embedding based on CLIP. This allows us to learn visual correspondences such as matching color and style, and produce more functionally plausible and aesthetically pleasing scenes. Evaluated on the 3D-FRONT dataset, our model achieves SOTA results in scene synthesis and improves auto-completion metrics by over 50%. Moreover, our embedding-based approach enables zero-shot text-guided scene synthesis and editing, which easily generalizes to furniture not seen during training.