mlsanityData to 7 Oct 2026

Training-Free Layout Control with Cross-Attention Guidance

arXiv 2304.03373 ↗First posted 6 Apr 2023cs.CVcited by 312

Not on the map. Why?

Cited by 185 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent diffusion-based generators can produce high-quality images from textual prompts. However, they often disregard textual instructions that specify the spatial layout of the composition. We propose a simple approach that achieves robust layout control without the need for training or fine-tuning of the image generator. Our technique manipulates the cross-attention layers that the model uses to interface textual and visual information and steers the generation in the desired direction given, e.g., a user-specified layout. To determine how to best guide attention, we study the role of attention maps and explore two alternative strategies, forward and backward guidance. We thoroughly evaluate our approach on three benchmarks and provide several qualitative examples and a comparative analysis of the two strategies that demonstrate the superiority of backward guidance compared to forward guidance, as well as prior work. We further demonstrate the versatility of layout guidance by extending it to applications such as editing the layout and context of real images.

Key facts. Cited by 312 papers in the map (data through 7 Oct 2026). Builds on 55 of its 63 references that are in the map. First posted on arXiv 6 Apr 2023.

Builds on

55 of 63 references.

References · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  8. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  9. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  10. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  11. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  12. 2016Generative Adversarial Text to Image Synthesiscited by 735
  13. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  14. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  15. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  16. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  17. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  18. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  19. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  20. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  21. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  22. 2023Universal Guidance for Diffusion Modelscited by 426
  23. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  24. 2018Image Generation from Scene Graphscited by 324
  25. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  26. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  27. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  28. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  29. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  30. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  31. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  32. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
  33. 2018Image Generation from Layoutcited by 105
  34. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  35. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  36. 2019Image Synthesis From Reconfigurable Layout and Stylecited by 85
  37. 2021Multimodal Conditional Image Synthesis with Product-of-Experts GANscited by 75
  38. 2022Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesiscited by 75

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 47; 2024: 126; 2025: 95; 2026: 44.

312 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  3. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  4. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  5. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 280
  6. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  7. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  8. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  9. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  10. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  11. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  12. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
38 more
  1. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  2. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 163
  3. 2023Dense Text-to-Image Generation with Attention Modulationcited by 162
  4. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  5. 2024Implicit Style-Content Separation using B-LoRAcited by 135
  6. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 132
  7. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
  8. 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
  9. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 95
  10. 2024DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editingcited by 92
  11. 2023FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Conditioncited by 89
  12. 2023LLM-grounded Video Diffusion Modelscited by 83
  13. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Modelscited by 82
  14. 2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
  15. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
  16. 2024DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generationcited by 73
  17. 2023TaleCrafter: Interactive Story Visualization with Multiple Characterscited by 71
  18. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 68
  19. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  20. 2023Zero-shot spatial layout conditioning for text-to-image diffusion modelscited by 64
  21. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60
  22. 2023Motion-Guided Latent Diffusion for Temporally Consistent Real-world Video Super-resolutioncited by 59
  23. 2023Diffusion Handles: Enabling 3D Edits for Diffusion Models by Lifting Activations to 3Dcited by 59
  24. 2024Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidancecited by 58
  25. 2023ControlRoom3D: Room Generation using Semantic Proxy Roomscited by 54
  26. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Promptscited by 53
  27. 2023Readout Guidance: Learning Control from Diffusion Featurescited by 52
  28. 2023WOUAF: Weight Modulation for User Attribution and Fingerprinting in Text-to-Image Diffusion Modelscited by 51
  29. 2024MagicTime: Time-lapse Video Generation Models as Metamorphic Simulatorscited by 51
  30. 2024VideoTetris: Towards Compositional Text-to-Video Generationcited by 51
  31. 2025ConceptAttention: Diffusion Transformers Learn Highly Interpretable Featurescited by 48
  32. 2024StyleMaster: Stylize Your Video with Artistic Generation and Translationcited by 47
  33. 2024DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusioncited by 46
  34. 2023DGInStyle: Domain-Generalizable Semantic Segmentation with Image Diffusion Models and Stylized Semantic Controlcited by 44
  35. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  36. 2024SceneCraft: Layout-Guided 3D Scene Generationcited by 40
  37. 2024HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generationcited by 39
  38. 2023Amodal Completion via Progressive Mixed Context Diffusioncited by 38

50 most cited of 312 All 312 (JSON)