mlsanityData to 7 Oct 2026

Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models

arXiv 2305.13921 ↗First posted 23 May 2023cs.CVcited by 82

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent text-to-image (T2I) diffusion models show outstanding performance in generating high-quality images conditioned on textual prompts. However, they fail to semantically align the generated images with the prompts due to their limited compositional capabilities, leading to attribute leakage, entity leakage, and missing entities. In this paper, we propose a novel attention mask control strategy based on predicted object boxes to address these issues. In particular, we first train a BoxNet to predict a box for each entity that possesses the attribute specified in the prompt. Then, depending on the predicted boxes, a unique mask control is applied to the cross- and self-attention maps. Our approach produces a more semantically accurate synthesis by constraining the attention regions of each token in the prompt to the image. In addition, the proposed method is straightforward and effective and can be readily integrated into existing cross-attention-based T2I generators. We compare our approach to competing methods and demonstrate that it can faithfully convey the semantics of the original text to the generated content and achieve high availability as a ready-to-use plugin. Please refer to https://github.com/OPPOMente-Lab/attention-mask-control.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 31 of its 35 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

31 of 35 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis7 Apr
  2. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuning20 Mar
  3. 2023Cones: Concept Neurons in Diffusion Models for Customized Generation9 Mar
  4. 2023Directed Diffusion: Direct Control of Object Placement through Attention Guidance25 Feb
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  6. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  7. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  8. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  9. 2022Shape-Guided Diffusion with Inside-Outside Attention1 Dec
  10. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  11. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
1 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2023Segment Anythingcited by 8,158
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2022Generative Adversarial Networkscited by 5,603
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
6 more
  1. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  2. 2022Diffusion Models in Vision: A Surveycited by 1,008
  3. 2016Tutorial on Variational Autoencoderscited by 404
  4. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  5. 2015End-to-end people detection in crowded scenescited by 140
  6. 2023Mixture of Diffusers for scene composition and high resolution image generationcited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 37; 2025: 20; 2026: 11.

82 papers.

Sort:

By area:

  1. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  2. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  3. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  4. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  5. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  6. 2023Improving Tuning-Free Real Image Editing with Proximal Guidancecited by 87
  7. 2024Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidancecited by 58
  8. 2024VideoTetris: Towards Compositional Text-to-Video Generationcited by 51
  9. 2023Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Modelscited by 36
  10. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  11. 2024Attention Calibration for Disentangled Text-to-Image Personalizationcited by 31
  12. 2024Training-free Regional Prompting for Diffusion Transformerscited by 26
38 more
  1. 2025CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion Modelscited by 26
  2. 2024AID: Attention Interpolation of Text-to-Image Diffusioncited by 24
  3. 2023A Survey of Diffusion Based Image Generation Models: Issues and Their Solutionscited by 23
  4. 2024CPR: Retrieval Augmented Generation for Copyright Protectioncited by 22
  5. 2023LoCo: Locally Constrained Training-Free Layout-to-Image Synthesiscited by 21
  6. 2023Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Modelscited by 19
  7. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  8. 2024Improving Compositional Attribute Binding in Text-to-Image Generative Models via Enhanced Text Embeddingscited by 18
  9. 2023MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Maskcited by 17
  10. 2025Multi-turn Consistent Image Editingcited by 17
  11. 2023Training Data Protection with Compositional Diffusion Modelscited by 16
  12. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decompositioncited by 16
  13. 2023Correcting Diffusion Generation through Resamplingcited by 15
  14. 2023Planning and Rendering: Towards Product Poster Generation with Diffusion Modelscited by 15
  15. 2024GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaborationcited by 15
  16. 2023MoEController: Instruction-based Arbitrary Image Manipulation with Mixture-of-Expert Controllerscited by 12
  17. 2023A Contrastive Compositional Benchmark for Text-to-Image Synthesis: A Study with Unified Text-to-Image Fidelity Metricscited by 12
  18. 2024Scene Graph Disentanglement and Composition for Generalizable Complex Image Generationcited by 11
  19. 2024Towards Better Text-to-Image Generation Alignment via Attention Modulationcited by 9
  20. 2024Semantic Score Distillation Sampling for Compositional Text-to-3D Generationcited by 9
  21. 2024Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!cited by 8
  22. 2025FreeBlend: Advancing Concept Blending with Staged Feedback-Driven Interpolation Diffusioncited by 8
  23. 2025Conceptrol: Concept Control of Zero-shot Personalized Image Generationcited by 8
  24. 2024Generation of Complex 3D Human Motion by Temporal and Spatial Composition of Diffusion Modelscited by 7
  25. 2025Leveraging Semantic Attribute Binding for Free-Lunch Color Control in Diffusion Modelscited by 7
  26. 2025VSC: Visual Search Compositional Text-to-Image Diffusion Modelcited by 7
  27. 2025StyleGuard: Preventing Text-to-Image-Model-based Style Mimicry Attacks by Style Perturbationscited by 7
  28. 2024Enhancing Prompt Following with Visual Control Through Training-Free Mask-Guided Diffusioncited by 6
  29. 2024AttenCraft: Attention-guided Disentanglement of Multiple Concepts for Text-to-Image Customizationcited by 6
  30. 2024Draw Like an Artist: Complex Scene Generation with Diffusion Model via Composition, Painting, and Retouchingcited by 6
  31. 2025Inversion-DPO: Precise and Efficient Post-Training for Diffusion Modelscited by 6
  32. 2025Geometrical Properties of Text Token Embeddings for Strong Semantic Binding in Text-to-Image Generationcited by 5
  33. 2024Divide and Conquer: Language Models can Plan and Self-Correct for Compositional Text-to-Image Generationcited by 4
  34. 2024Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversioncited by 4
  35. 2024Diffusion Beats Autoregressive: An Evaluation of Compositional Generation in Text-to-Image Modelscited by 4
  36. 2025PositionIC: Unified Position and Identity Consistency for Image Customizationcited by 4
  37. 2024TextCenGen: Attention-Guided Text-Centric Background Adaptation for Text-to-Image Generationcited by 3
  38. 2024Improving Text Generation on Images with Synthetic Captionscited by 3

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models
  2. 2023Masked-Attention Diffusion Guidance for Spatially Controlling Text-to-Image Generation
  3. 2023Grounded Text-to-Image Synthesis with Attention Refocusing
  4. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
  5. 2024Layout-to-Image Generation with Localized Descriptions using ControlNet with Cross-Attention Control
  6. 2025UNCAGE: Contrastive Attention Guidance for Masked Generative Transformers in Text-to-Image Generation
  7. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
  8. 2025Hierarchical and Step-Layer-Wise Tuning of Attention Specialty for Multi-Instance Synthesis in Diffusion Transformers