mlsanityData to 7 Oct 2026

LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation

arXiv 2303.17189 ↗First posted 30 Mar 2023cs.CVcited by 258

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recently, diffusion models have achieved great success in image synthesis. However, when it comes to the layout-to-image generation where an image often has a complex scene of multiple objects, how to make strong control over both the global layout map and each detailed object remains a challenging task. In this paper, we propose a diffusion model named LayoutDiffusion that can obtain higher generation quality and greater controllability than the previous works. To overcome the difficult multimodal fusion of image and layout, we propose to construct a structural image patch with region information and transform the patched image into a special layout to fuse with the normal layout in a unified form. Moreover, Layout Fusion Module (LFM) and Object-aware Cross Attention (OaCA) are proposed to model the relationship among multiple objects and designed to be object-aware and position-sensitive, allowing for precisely controlling the spatial related information. Extensive experiments show that our LayoutDiffusion outperforms the previous SOTA methods on FID, CAS by relatively 46.35%, 26.70% on COCO-stuff and 44.29%, 41.82% on VG. Code is available at https://github.com/ZGCTroy/LayoutDiffusion.

Key facts. Cited by 258 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 49 of its 59 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

49 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  12. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
35 more
  1. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  3. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  4. 2016Improved Techniques for Training GANscited by 2,938
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  7. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  8. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  9. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  10. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  11. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  12. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  13. 2017Mixed Precision Trainingcited by 1,135
  14. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  15. 2022Diffusion Models in Vision: A Surveycited by 1,008
  16. 2022Denoising Diffusion Restoration Modelscited by 951
  17. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  18. 2018Panoptic Segmentationcited by 796
  19. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 460
  20. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 331
  21. 2018Image Generation from Scene Graphscited by 324
  22. 2019Classification Accuracy Score for Conditional Generative Modelscited by 134
  23. 2016Learning What and Where to Drawcited by 105
  24. 2018Image Generation from Layoutcited by 105
  25. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  26. 2019Image Synthesis From Reconfigurable Layout and Stylecited by 85
  27. 2019Specifying Object Attributes and Relations in Interactive Scene Generationcited by 76
  28. 2022Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Trainingcited by 56
  29. 2022Efficient Diffusion Models for Vision: A Surveycited by 50
  30. 2020Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesiscited by 49
  31. 2021Recent Advances and Trends in Multimodal Deep Learning: A Reviewcited by 26
  32. 2016Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layoutscited by 21
  33. 2022Entropy-driven Sampling and Training Scheme for Conditional Diffusion Generationcited by 20
  34. 2022Interactive Image Synthesis with Panoptic Layout Generationcited by 17
  35. 2018Text2Scene: Generating Compositional Scenes from Textual Descriptionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 23; 2024: 91; 2025: 100; 2026: 44.

258 papers.

Sort:

By area:

  1. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  2. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  3. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  4. 2024In-Context LoRA for Diffusion Transformerscited by 155
  5. 2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 142
  6. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
  7. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 115
  8. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  9. 2023Elucidating the Exposure Bias in Diffusion Modelscited by 98
  10. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  11. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generationcited by 92
  12. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
38 more
  1. 2024DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splattingcited by 81
  2. 2023DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion modelcited by 75
  3. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 68
  4. 2023Zero-shot spatial layout conditioning for text-to-image diffusion modelscited by 64
  5. 2024CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilitiescited by 58
  6. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 58
  7. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 54
  8. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Promptscited by 53
  9. 2024HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generationcited by 39
  10. 2024Compositional Text-to-Image Generation with Dense Blob Representationscited by 38
  11. 2025ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Modelscited by 37
  12. 2025PosterCraft: Rethinking High-Quality Aesthetic Poster Generation in a Unified Frameworkcited by 37
  13. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  14. 2024MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesiscited by 35
  15. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  16. 2024Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidancecited by 33
  17. 2025ViMo: A Generative Visual GUI World Model for App Agentscited by 33
  18. 2024DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perceptioncited by 32
  19. 2024AeroGen: Enhancing Remote Sensing Object Detection with Diffusion-Driven Data Generationcited by 32
  20. 2025EliGen: Entity-Level Controlled Image Generation with Regional Attentioncited by 32
  21. 2024All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seedscited by 31
  22. 2023SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generationcited by 29
  23. 2023R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generationcited by 29
  24. 2024LayerDiff: Exploring Text-guided Multi-layered Composable Image Synthesis via Layer-Collaborative Diffusion Modelcited by 26
  25. 2024GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantationcited by 24
  26. 2023The Lottery Ticket Hypothesis in Denoising: Towards Semantic-Driven Initializationcited by 22
  27. 2024DiffUHaul: A Training-Free Method for Object Dragging in Imagescited by 22
  28. 2023High-fidelity Person-centric Subject-to-Image Synthesiscited by 21
  29. 2024SG-Adapter: Enhancing Text-to-Image Generation with Scene Graph Guidancecited by 20
  30. 2024AnyControl: Create Your Artwork with Versatile Control on Text-to-Image Generationcited by 20
  31. 2023Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generationcited by 19
  32. 2023Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Modelscited by 19
  33. 2024AutoStudio: Crafting Consistent Subjects in Multi-turn Interactive Image Generationcited by 19
  34. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  35. 2024TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generationcited by 18
  36. 2024The Crystal Ball Hypothesis in diffusion models: Anticipating object positions from initial noisecited by 18
  37. 2024Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generationcited by 18
  38. 2024IFAdapter: Instance Feature Control for Grounded Text-to-Image Generationcited by 18

50 most cited of 258 All 258 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Layered Rendering Diffusion Model for Controllable Zero-Shot Image Synthesis
  2. 2023LAW-Diffusion: Complex Scene Generation by Diffusion with Layouts
  3. 2025A Two-Stage System for Layout-Controlled Image Generation using Large Language Models and Diffusion Models
  4. 2023LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation
  5. 2024Layout Control and Semantic Guidance with Attention Loss Backward for T2I Diffusion Model
  6. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation
  7. 2023Unifying Layout Generation with a Decoupled Diffusion Model
  8. 2023LayoutDiffusion: Improving Graphic Layout Generation by Discrete Diffusion Probabilistic Models