# LAW-Diffusion: Complex Scene Generation by Diffusion with Layouts

arXiv: https://arxiv.org/abs/2308.06713 · HTML: https://mlsanity.com/paper/2308.06713/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 39 of its 49 references that are in the map.
- First posted on arXiv 13 Aug 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [ReCo: Region-Controlled Text-to-Image Generation](https://mlsanity.com/paper/2211.15518/) (2022)
- [Classification Accuracy Score for Conditional Generative Models](https://mlsanity.com/paper/1905.10887/) (2019)

## Built on by (most cited first, 20 of 20)

- [CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation](https://mlsanity.com/paper/2412.03859/) (2024)
- [CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Design](https://mlsanity.com/paper/2505.19114/) (2025)
- [LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis](https://mlsanity.com/paper/2311.12342/) (2023)
- [FRDiff : Feature Reuse for Universal Training-free Acceleration of Diffusion Models](https://mlsanity.com/paper/2312.03517/) (2023)
- [AnyControl: Create Your Artwork with Versatile Control on Text-to-Image Generation](https://mlsanity.com/paper/2406.18958/) (2024)
- [Planning and Rendering: Towards Product Poster Generation with Diffusion Models](https://mlsanity.com/paper/2312.08822/) (2023)
- [iFusion: Inverting Diffusion for Pose-Free Reconstruction from Sparse Views](https://mlsanity.com/paper/2312.17250/) (2023)
- [MMGen: Unified Multi-modal Image Generation and Understanding in One Go](https://mlsanity.com/paper/2503.20644/) (2025)
- [Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control](https://mlsanity.com/paper/2506.00596/) (2025)
- [InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention](https://mlsanity.com/paper/2509.16691/) (2025)
- [DiffRefiner: Coarse to Fine Trajectory Planning via Diffusion Refinement with Semantic Interaction for End to End Autonomous Driving](https://mlsanity.com/paper/2511.17150/) (2025)
- [Scene123: One Prompt to 3D Scene Generation via Video-Assisted and Consistency-Enhanced MAE](https://mlsanity.com/paper/2408.05477/) (2024)
- [Draw Like an Artist: Complex Scene Generation with Diffusion Model via Composition, Painting, and Retouching](https://mlsanity.com/paper/2408.13858/) (2024)
- [Diffusion Model Compression for Image-to-Image Translation](https://mlsanity.com/paper/2401.17547/) (2024)
- [DiffX: Guide Your Layout to Cross-Modal Generative Modeling](https://mlsanity.com/paper/2407.15488/) (2024)
- [Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospects](https://mlsanity.com/paper/2410.01816/) (2024)
- [Cross-Subject Mind Decoding from Inaccurate Representations](https://mlsanity.com/paper/2507.19071/) (2025)
- [Appearance Pointers -- Multimodal Region Control of Diffusion Transformers](https://mlsanity.com/paper/2607.19344/) (2026)
- [ControlRef: Efficient Layout-Guided Multi-Instance Generation via Anchored 4D-RoPE](https://mlsanity.com/paper/2608.06878/) (2026)
- [Bridging Language and Spherical Space: Object-Centric Control for Text-to-Panorama Generation](https://mlsanity.com/paper/2608.20691/) (2026)

## Abstract (arXiv)

Thanks to the rapid development of diffusion models, unprecedented progress has been witnessed in image synthesis. Prior works mostly rely on pre-trained linguistic models, but a text is often too abstract to properly specify all the spatial properties of an image, e.g., the layout configuration of a scene, leading to the sub-optimal results of complex scene generation. In this paper, we achieve accurate complex scene generation by proposing a semantically controllable Layout-AWare diffusion model, termed LAW-Diffusion. Distinct from the previous Layout-to-Image generation (L2I) methods that only explore category-aware relationships, LAW-Diffusion introduces a spatial dependency parser to encode the location-aware semantic coherence across objects as a layout embedding and produces a scene with perceptually harmonious object styles and contextual relations. To be specific, we delicately instantiate each object's regional semantics as an object region map and leverage a location-aware cross-object attention module to capture the spatial dependencies among those disentangled representations. We further propose an adaptive guidance schedule for our layout guidance to mitigate the trade-off between the regional semantic alignment and the texture fidelity of generated objects. Moreover, LAW-Diffusion allows for instance reconfiguration while maintaining the other regions in a synthesized image by introducing a layout-aware latent grafting mechanism to recompose its local regional semantics. To better verify the plausibility of generated scenes, we propose a new evaluation metric for the L2I task, dubbed Scene Relation Score (SRS) to measure how the images preserve the rational and harmonious relations among contextual objects. Comprehensive experiments demonstrate that our LAW-Diffusion yields the state-of-the-art generative performance, especially with coherent object relations.