mlsanityData to 7 Oct 2026

Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis

arXiv 2212.05032 ↗First posted 9 Dec 2022cs.CV cs.CLcited by 386

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Large-scale diffusion models have achieved state-of-the-art results on text-to-image synthesis (T2I) tasks. Despite their ability to generate high-quality yet creative images, we observe that attribution-binding and compositional capabilities are still considered major challenging issues, especially when involving multiple objects. In this work, we improve the compositional skills of T2I models, specifically more accurate attribute binding and better image compositions. To do this, we incorporate linguistic structures with the diffusion guidance process based on the controllable properties of manipulating cross-attention layers in diffusion-based T2I models. We observe that keys and values in cross-attention layers have strong semantic meanings associated with object layouts and content. Therefore, we can better preserve the compositional semantics in the generated image by manipulating the cross-attention representations based on linguistic insights. Built upon Stable Diffusion, a SOTA T2I model, our structured cross-attention design is efficient that requires no additional training samples. We achieve better compositional skills in qualitative and quantitative results, leading to a 5-8% advantage in head-to-head user comparison studies. Lastly, we conduct an in-depth analysis to reveal potential causes of incorrect image compositions and justify the properties of cross-attention layers in the generation process.

Key facts. Cited by 386 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 39 of its 51 references that are in the map. First posted on arXiv 9 Dec 2022.

Builds on

39 of 51 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  2. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  6. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance10 Dec

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  11. 2016Improved Techniques for Training GANscited by 2,938
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
21 more
  1. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  2. 2021Grounded Language-Image Pre-trainingcited by 1,147
  3. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  4. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  5. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  6. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  7. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  8. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  9. 2018Image Generation from Scene Graphscited by 324
  10. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  11. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  12. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  13. 2019Controllable Text-to-Image Generationcited by 187
  14. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
  15. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  16. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  17. 2018Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instructioncited by 61
  18. 2020Comprehensive Image Captioning via Scene Graph Decompositioncited by 52
  19. 2022Modeling Image Composition for Complex Scene Generationcited by 43
  20. 2020SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoningcited by 32
  21. 2022Unsupervised Vision-Language Parsing: Seamlessly Bridging Visual Scene Graphs with Language Structures via Dependency Relationshipscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 113; 2024: 147; 2025: 85; 2026: 40.

386 papers.

Sort:

By area:

  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  3. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  4. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  5. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  6. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  7. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 486
  8. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  9. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  10. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  11. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  12. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
38 more
  1. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  2. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  3. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  4. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Modelcited by 290
  5. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 284
  6. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  7. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 280
  8. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  9. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  10. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 240
  11. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  12. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 193
  13. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  14. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  15. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  16. 2023Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignmentcited by 169
  17. 2023UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANscited by 165
  18. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 163
  19. 2023Dense Text-to-Image Generation with Attention Modulationcited by 162
  20. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  21. 2023Cones: Concept Neurons in Diffusion Models for Customized Generationcited by 150
  22. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Codecited by 145
  23. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 140
  24. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 132
  25. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
  26. 2024InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimizationcited by 123
  27. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  28. 2023Cones 2: Customizable Image Synthesis with Multiple Subjectscited by 101
  29. 2024Video Diffusion Alignment via Reward Gradientscited by 101
  30. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  31. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  32. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Modelscited by 94
  33. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generationcited by 92
  34. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  35. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  36. 2024DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editingcited by 88
  37. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 88
  38. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 85

50 most cited of 386 All 386 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
  2. 2024Training-free Composite Scene Generation for Layout-to-Image Synthesis
  3. 2024All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds
  4. 2023KOALA: Empirical Lessons Toward Memory-Efficient and Fast Diffusion Models for Text-to-Image Synthesis
  5. 2023SEGA: Instructing Text-to-Image Models using Semantic Guidance
  6. 2026ORCA: Hunting Compositional Failures in Text-to-Image Diffusion
  7. 2024TALE: Training-free Cross-domain Image Composition via Adaptive Latent Manipulation and Energy-guided Optimization
  8. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models