mlsanityData to 7 Oct 2026

LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation

arXiv 2302.08908 ↗First posted 16 Feb 2023cs.CVcited by 65

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Layout-to-image generation refers to the task of synthesizing photo-realistic images based on semantic layouts. In this paper, we propose LayoutDiffuse that adapts a foundational diffusion model pretrained on large-scale image or text-image datasets for layout-to-image generation. By adopting a novel neural adaptor based on layout attention and task-aware prompts, our method trains efficiently, generates images with both high perceptual quality and layout alignment, and needs less data. Experiments on three datasets show that our method significantly outperforms other 10 generative models based on GANs, VQ-VAE, and diffusion models.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 43 of its 47 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

43 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  5. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
26 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  4. 2016Improved Techniques for Training GANscited by 2,938
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  8. 2022Visual Prompt Tuningcited by 1,674
  9. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  12. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  13. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  14. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  15. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  16. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  17. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  18. 2018Image Generation from Scene Graphscited by 324
  19. 2018Image Generation from Layoutcited by 105
  20. 2022DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latentscited by 103
  21. 2019Image Synthesis From Reconfigurable Layout and Stylecited by 85
  22. 2020Object-Centric Image Generation from Layoutscited by 57
  23. 2022Modeling Image Composition for Complex Scene Generationcited by 43
  24. 2021Context-Aware Layout to Image Generation with Enhanced Object Appearancecited by 41
  25. 2021High-Resolution Complex Scene Synthesis with Transformerscited by 31
  26. 2022Decoupled Multi-task Learning with Cyclical Self-Regulation for Face Parsingcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 33; 2025: 16; 2026: 3.

65 papers.

Sort:

By area:

  1. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  2. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  3. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  4. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generationcited by 92
  5. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  6. 2023Consistent Video-to-Video Transfer Using Synthetic Datasetcited by 79
  7. 2023DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion modelcited by 75
  8. 2023GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generationcited by 56
  9. 2024HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generationcited by 39
  10. 2024SimGen: Simulator-conditioned Driving Scene Generationcited by 34
  11. 2024DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perceptioncited by 32
  12. 2024NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Mergingcited by 25
38 more
  1. 2024GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantationcited by 24
  2. 2023The Lottery Ticket Hypothesis in Denoising: Towards Semantic-Driven Initializationcited by 22
  3. 2023LoCo: Locally Constrained Training-Free Layout-to-Image Synthesiscited by 21
  4. 2024AnyControl: Create Your Artwork with Versatile Control on Text-to-Image Generationcited by 20
  5. 2024ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidancecited by 18
  6. 2024IFAdapter: Instance Feature Control for Grounded Text-to-Image Generationcited by 18
  7. 2023MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Maskcited by 17
  8. 2024ODGEN: Domain-specific Object Detection Data Generation with Diffusion Modelscited by 17
  9. 2024FocusDiffuser: Perceiving Local Disparities for Camouflaged Object Detectioncited by 14
  10. 2024The Silent Assistant: NoiseQuery as Implicit Guidance for Goal-Driven Image Generationcited by 14
  11. 2024Scene Graph Disentanglement and Composition for Generalizable Complex Image Generationcited by 11
  12. 2023Layered Rendering Diffusion Model for Controllable Zero-Shot Image Synthesiscited by 9
  13. 2024Towards Better Text-to-Image Generation Alignment via Attention Modulationcited by 9
  14. 2025InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attentioncited by 9
  15. 2024Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generationcited by 8
  16. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  17. 2025PixelPonder: Dynamic Patch Adaptation for Enhanced Multi-Conditional Text-to-Image Generationcited by 7
  18. 2025QuaDreamer: Controllable Panoramic Video Generation for Quadruped Robotscited by 7
  19. 2024When ControlNet Meets Inexplicit Masks: A Case Study of ControlNet on its Contour-following Abilitycited by 6
  20. 2024ReGround: Improving Textual and Spatial Grounding at No Costcited by 6
  21. 2024Isolated Diffusion: Optimizing Multi-Concept Text-to-Image Generation Training-Freely with Isolated Diffusion Guidancecited by 6
  22. 2024Enhancing Image Layout Control with Loss-Guided Diffusion Modelscited by 6
  23. 2024MyGo: Consistent and Controllable Multi-View Driving Video Generation with Camera Controlcited by 6
  24. 2025ReCon: Region-Controllable Data Augmentation with Rectification and Alignment for Object Detectioncited by 6
  25. 2024Panacea+: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 5
  26. 2023Identifying Systematic Errors in Object Detectors with the SCROD Pipelinecited by 4
  27. 2024Joint Generative Modeling of Grounded Scene Graphs and Images via Diffusion Modelscited by 4
  28. 2024DODA: Adapting Object Detectors to Dynamic Agricultural Environments in Real-Time with Diffusioncited by 4
  29. 2024Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesiscited by 4
  30. 2025FICGen: Frequency-Inspired Contextual Disentanglement for Layout-driven Degraded Image Generationcited by 4
  31. 2025Unbiased Object Detection Beyond Frequency with Visually Prompted Image Synthesiscited by 4
  32. 2024Dense-Face: Personalized Face Generation Model via Dense Annotation Predictioncited by 3
  33. 2025STAY Diffusion: Styled Layout Diffusion Model for Diverse Layout-to-Image Generationcited by 3
  34. 2025Local Mechanisms of Compositional Generalization in Conditional Diffusioncited by 3
  35. 2024Strictly-ID-Preserved and Controllable Accessory Advertising Image Generationcited by 2
  36. 2024Financial Models in Generative Art: Black-Scholes-Inspired Concept Blending in Text-to-Image Diffusioncited by 2
  37. 2024Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospectscited by 2
  38. 2025CoProSketch: Controllable and Progressive Sketch Generation with Diffusion Modelcited by 2

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Training-free Composite Scene Generation for Layout-to-Image Synthesis
  2. 2023Reason out Your Layout: Evoking the Layout Master from Large Language Models for Text-to-Image Synthesis
  3. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation
  4. 2023LayoutDM: Transformer-based Diffusion Model for Layout Generation
  5. 2024LayoutDiT: Exploring Content-Graphic Balance in Layout Generation with Diffusion Transformer
  6. 2025A Two-Stage System for Layout-Controlled Image Generation using Large Language Models and Diffusion Models
  7. 2024Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive
  8. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation