mlsanityData to 7 Oct 2026

Grounded Text-to-Image Synthesis with Attention Refocusing

arXiv 2306.05427 ↗First posted 8 Jun 2023cs.CVcited by 163

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Driven by the scalable diffusion models trained on large-scale datasets, text-to-image synthesis methods have shown compelling results. However, these models still fail to precisely follow the text prompt involving multiple objects, attributes, or spatial compositions. In this paper, we reveal the potential causes in the diffusion model's cross-attention and self-attention layers. We propose two novel losses to refocus attention maps according to a given spatial layout during sampling. Creating the layouts manually requires additional effort and can be tedious. Therefore, we explore using large language models (LLM) to produce these layouts for our method. We conduct extensive experiments on the DrawBench, HRS, and TIFA benchmarks to evaluate our proposed method. We show that our proposed attention refocusing effectively improves the controllability of existing approaches.

Key facts. Cited by 163 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 54 of its 66 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

54 of 66 references.

Closest ancestors · 18

Same direction, newest first

  1. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models23 May
  2. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention17 May
  3. 2023SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models9 May
  4. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models11 Apr
  5. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis7 Apr
  6. 2023Directed Diffusion: Direct Control of Object Placement through Attention Guidance25 Feb
  7. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  9. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  10. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  11. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  12. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
6 more
  1. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  2. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2020Denoising Diffusion Implicit Modelscited by 9,655
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
24 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  5. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  6. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  7. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  8. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  9. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  10. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  11. 2023Universal Guidance for Diffusion Modelscited by 426
  12. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  13. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  14. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  15. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  16. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  17. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  18. 2023Teaching CLIP to Count to Tencited by 194
  19. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
  20. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  21. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  22. 2023Mixture of Diffusers for scene composition and high resolution image generationcited by 56
  23. 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 53
  24. 2023SimpSON: Simplifying Photo Cleanup with Single-Click Distracting Object Segmentation Networkcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 22; 2024: 67; 2025: 48; 2026: 26.

163 papers.

Sort:

By area:

  1. 2024MACE: Mass Concept Erasure in Diffusion Modelscited by 298
  2. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 280
  3. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  4. 2023Dense Text-to-Image Generation with Attention Modulationcited by 162
  5. 2023LLM-grounded Video Diffusion Modelscited by 83
  6. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  7. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  8. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 68
  9. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  10. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  11. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60
  12. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 58
38 more
  1. 2024LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Modelscited by 55
  2. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Promptscited by 53
  3. 2025VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editingcited by 50
  4. 2024Make It Count: Text-to-Image Generation with an Accurate Number of Objectscited by 47
  5. 2024MultiBooth: Towards Generating All Your Concepts in an Image from Textcited by 45
  6. 2024Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Modelscited by 45
  7. 2024SceneTeller: Language-to-3D Scene Generationcited by 43
  8. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generationcited by 40
  9. 2023AutoStory: Generating Diverse Storytelling Images with Minimal Human Effortcited by 38
  10. 2024Compositional Text-to-Image Generation with Dense Blob Representationscited by 38
  11. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  12. 2023MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentationcited by 35
  13. 2024PALP: Prompt Aligned Personalization of Text-to-Image Modelscited by 35
  14. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  15. 2024Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Surveycited by 34
  16. 2023R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generationcited by 29
  17. 2025DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale Text-to-Image Modelscited by 28
  18. 2024Learning Flow Fields in Attention for Controllable Person Image Generationcited by 27
  19. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  20. 2024NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Mergingcited by 25
  21. 2023Counting Guidance for High Fidelity Text-to-Image Synthesiscited by 24
  22. 2023ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generationscited by 24
  23. 2024SmartControl: Enhancing ControlNet for Handling Rough Visual Conditionscited by 24
  24. 2024GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantationcited by 24
  25. 2023A Survey of Diffusion Based Image Generation Models: Issues and Their Solutionscited by 23
  26. 2023LoCo: Locally Constrained Training-Free Layout-to-Image Synthesiscited by 21
  27. 2023MAG-Edit: Localized Image Editing in Complex Scenarios via Mask-Based Attention-Adjusted Guidancecited by 21
  28. 2025The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generationcited by 21
  29. 2024PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesiscited by 20
  30. 2023Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generationcited by 19
  31. 2024A Cat Is A Cat (Not A Dog!): Unraveling Information Mix-ups in Text-to-Image Encoders through Causal Analysis and Embedding Optimizationcited by 19
  32. 2025Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformerscited by 19
  33. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  34. 2024MaGGIe: Masked Guided Gradual Human Instance Mattingcited by 18
  35. 2024The Crystal Ball Hypothesis in diffusion models: Anticipating object positions from initial noisecited by 18
  36. 2025BizGen: Advancing Article-level Visual Text Rendering for Infographics Generationcited by 18
  37. 2023MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Maskcited by 17
  38. 2024EVA: Zero-shot Accurate Attributes and Multi-Object Video Editingcited by 17

50 most cited of 163 All 163 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models
  2. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models
  3. 2023Masked-Attention Diffusion Guidance for Spatially Controlling Text-to-Image Generation
  4. 2023Fast Sampling Through The Reuse Of Attention Maps In Diffusion Models
  5. 2025Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesis
  6. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
  7. 2024CoCoNO: Attention Contrast-and-Complete for Initial Noise Optimization in Text-to-Image Synthesis
  8. 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models