mlsanityData to 7 Oct 2026

Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

arXiv 2301.13826 ↗First posted 31 Jan 2023cs.CV cs.CL cs.GRcited by 696

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent text-to-image generative models have demonstrated an unparalleled ability to generate diverse and creative imagery guided by a target text prompt. While revolutionary, current state-of-the-art diffusion models may still fail in generating images that fully convey the semantics in the given text prompt. We analyze the publicly available Stable Diffusion model and assess the existence of catastrophic neglect, where the model fails to generate one or more of the subjects from the input prompt. Moreover, we find that in some cases the model also fails to correctly bind attributes (e.g., colors) to their corresponding subjects. To help mitigate these failure cases, we introduce the concept of Generative Semantic Nursing (GSN), where we seek to intervene in the generative process on the fly during inference time to improve the faithfulness of the generated images. Using an attention-based formulation of GSN, dubbed Attend-and-Excite, we guide the model to refine the cross-attention units to attend to all subject tokens in the text prompt and strengthen - or excite - their activations, encouraging the model to generate all subjects described in the text prompt. We compare our approach to alternative approaches and demonstrate that it conveys the desired concepts more faithfully across a range of text prompts.

Key facts. Cited by 696 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 53 of its 62 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

53 of 62 references.

Closest ancestors · 16

Same direction, newest first

  1. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  2. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
  3. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  4. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  5. 2022Investigating Prompt Engineering in Diffusion Models21 Nov
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  7. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  8. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models26 Oct
  9. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  10. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  12. 2022Blended Latent Diffusion6 Jun
4 more
  1. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
25 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  2. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  3. 2020Quantifying Attention Flow in Transformerscited by 759
  4. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  5. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  6. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  7. 2018Image Generation from Scene Graphscited by 324
  8. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  9. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  10. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  11. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  12. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  13. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  14. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  15. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
  16. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  17. 2021Paint by Wordcited by 107
  18. 2018Image Generation from Layoutcited by 105
  19. 2022A very preliminary analysis of DALL-E 2cited by 82
  20. 2019Specifying Object Attributes and Relations in Interactive Scene Generationcited by 76
  21. 2021Improving Text-to-Image Synthesis Using Contrastive Learningcited by 62
  22. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  23. 2022Optimizing Relevance Maps of Vision Transformers Improves Robustnesscited by 41
  24. 2021Image-Based CLIP-Guided Essence Transfercited by 39
  25. 2022No Token Left Behind: Explainability-Aided Image Classification and Generationcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 138; 2024: 249; 2025: 174; 2026: 135.

696 papers.

Sort:

By area:

  1. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  2. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  3. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  4. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  5. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  6. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  7. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  8. 2024MACE: Mass Concept Erasure in Diffusion Modelscited by 298
  9. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 297
  10. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  11. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 280
  12. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
38 more
  1. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  2. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  3. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  4. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  5. 2023Style Aligned Image Generation via Shared Attentioncited by 214
  6. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  7. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  8. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  9. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  10. 2023Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignmentcited by 169
  11. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  12. 2023UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANscited by 165
  13. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 163
  14. 2023Dense Text-to-Image Generation with Attention Modulationcited by 162
  15. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  16. 2023Cones: Concept Neurons in Diffusion Models for Customized Generationcited by 150
  17. 2023LEDITS++: Limitless Image Editing using Text-to-Image Modelscited by 147
  18. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Modelscited by 142
  19. 2023SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generationcited by 142
  20. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  21. 2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 135
  22. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 132
  23. 2023Editing Implicit Assumptions in Text-to-Image Diffusion Modelscited by 128
  24. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  25. 2023A Neural Space-Time Representation for Text-to-Image Personalizationcited by 123
  26. 2024Training-Free Consistent Text-to-Image Generationcited by 123
  27. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
  28. 2024InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimizationcited by 123
  29. 2023Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometrycited by 121
  30. 2025EasyControl: Adding Efficient and Flexible Control for Diffusion Transformercited by 118
  31. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  32. 2023Vox-E: Text-guided Voxel Editing of 3D Objectscited by 114
  33. 2024One-Step Image Translation with Text-to-Image Modelscited by 111
  34. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  35. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 104
  36. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
  37. 2023Cones 2: Customizable Image Synthesis with Multiple Subjectscited by 101
  38. 2023Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Modelscited by 98

50 most cited of 696 All 696 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023SEGA: Instructing Text-to-Image Models using Semantic Guidance
  2. 2023SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models
  3. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
  4. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models
  5. 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
  6. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models
  7. 2023DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation
  8. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models