mlsanityData to 7 Oct 2026

Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment

arXiv 2306.08877 ↗First posted 15 Jun 2023cs.CL cs.CVcited by 169

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-conditioned image generation models often generate incorrect associations between entities and their visual attributes. This reflects an impaired mapping between linguistic binding of entities and modifiers in the prompt and visual binding of the corresponding elements in the generated image. As one notable example, a query like "a pink sunflower and a yellow flamingo" may incorrectly produce an image of a yellow sunflower and a pink flamingo. To remedy this issue, we propose SynGen, an approach which first syntactically analyses the prompt to identify entities and their modifiers, and then uses a novel loss function that encourages the cross-attention maps to agree with the linguistic binding reflected by the syntax. Specifically, we encourage large overlap between attention maps of entities and their modifiers, and small overlap with other entities and modifier words. The loss is optimized during inference, without retraining or fine-tuning the model. Human evaluation on three datasets, including one new and challenging set, demonstrate significant improvements of SynGen compared with current state of the art methods. This work highlights how making use of sentence structure during inference can efficiently and substantially improve the faithfulness of text-to-image generation.

Key facts. Cited by 169 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 21 of its 24 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

21 of 24 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis7 Apr
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  3. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  4. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  5. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  6. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 13

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  7. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  8. 2021Open World Compositional Zero-Shot Learningcited by 100
  9. 2020A causal view of compositional zero-shot recognitioncited by 78
  10. 2022Testing Relational Understanding in Text-Guided Image Generationcited by 59
  11. 2022DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Modelscited by 50
  12. 2022DALL-E 2 Fails to Reliably Capture Common Syntactic Processescited by 31
1 more
  1. 2016Learning to generalize to new compositions in image understandingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 60; 2025: 54; 2026: 40.

169 papers.

Sort:

By area:

  1. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  2. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  3. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  4. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  5. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  6. 2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 135
  7. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  8. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  9. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 68
  10. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60
  11. 2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 57
  12. 2024Make It Count: Text-to-Image Generation with an Accurate Number of Objectscited by 47
38 more
  1. 2024TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generationcited by 42
  2. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generationcited by 40
  3. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  4. 2024Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directionscited by 39
  5. 2023Interpretable Diffusion via Information Decompositioncited by 37
  6. 2024PALP: Prompt Aligned Personalization of Text-to-Image Modelscited by 35
  7. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  8. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  9. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  10. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  11. 2025T2I-Copilot: A Training-Free Multi-Agent Text-to-Image System for Enhanced Prompt Interpretation and Interactive Generationcited by 31
  12. 2024Prioritized Semantic Learning for Zero-shot Instance Navigationcited by 30
  13. 2024CosmicMan: A Text-to-Image Foundation Model for Humanscited by 30
  14. 2024Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidancecited by 30
  15. 2024ColorPeel: Color Prompt Learning with Diffusion Models via Color and Shape Disentanglementcited by 28
  16. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Modelscited by 27
  17. 2024MC2: Multi-concept Guidance for Customized Multi-concept Generationcited by 26
  18. 2024AID: Attention Interpolation of Text-to-Image Diffusioncited by 24
  19. 2023A Survey of Diffusion Based Image Generation Models: Issues and Their Solutionscited by 23
  20. 2023MAG-Edit: Localized Image Editing in Complex Scenarios via Mask-Based Attention-Adjusted Guidancecited by 21
  21. 2023Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Modelscited by 19
  22. 2024A Cat Is A Cat (Not A Dog!): Unraveling Information Mix-ups in Text-to-Image Encoders through Causal Analysis and Embedding Optimizationcited by 19
  23. 2025Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformerscited by 19
  24. 2023Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignmentcited by 18
  25. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  26. 2024Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image Diffusion Modelscited by 18
  27. 2024ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidancecited by 18
  28. 2024Improving Compositional Attribute Binding in Text-to-Image Generative Models via Enhanced Text Embeddingscited by 18
  29. 2023Predicated Diffusion: Predicate Logic-Based Attention Guidance for Text-to-Image Diffusion Modelscited by 16
  30. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decompositioncited by 16
  31. 2024Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjectscited by 16
  32. 2024SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Datacited by 14
  33. 2024Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Modelscited by 14
  34. 2025Towards Transformer-Based Aligned Generation with Self-Coherence Guidancecited by 14
  35. 2024Compositional Video Generation as Flow Equalizationcited by 12
  36. 2024Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Mapscited by 12
  37. 2025Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Modelscited by 12
  38. 2025Side Effects of Erasing Concepts from Diffusion Modelscited by 12

50 most cited of 169 All 169 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Divide & Bind Your Attention for Improved Generative Semantic Nursing
  2. 2024Towards Better Text-to-Image Generation Alignment via Attention Modulation
  3. 2024Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function
  4. 2023Semantic Guidance Tuning for Text-To-Image Diffusion Models
  5. 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models
  6. 2026Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference
  7. 2023Localizing and Editing Knowledge in Text-to-Image Generative Models
  8. 2023The Hidden Language of Diffusion Models