mlsanityData to 7 Oct 2026

Dense Text-to-Image Generation with Attention Modulation

arXiv 2308.12964 ↗First posted 24 Aug 2023cs.CV cs.GR cs.LGcited by 162

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Existing text-to-image diffusion models struggle to synthesize realistic images given dense captions, where each text prompt provides a detailed description for a specific image region. To address this, we propose DenseDiffusion, a training-free method that adapts a pre-trained text-to-image model to handle such dense captions while offering control over the scene layout. We first analyze the relationship between generated images' layouts and the pre-trained model's intermediate attention maps. Next, we develop an attention modulation method that guides objects to appear in specific regions according to layout guidance. Without requiring additional fine-tuning or datasets, we improve image generation performance given dense captions regarding both automatic and human evaluation scores. In addition, we achieve similar-quality visual results with models specifically trained with layout conditions.

Key facts. Cited by 162 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 57 of its 60 references that are in the map. First posted on arXiv 24 Aug 2023.

Builds on

57 of 60 references.

Closest ancestors · 20

Same direction, newest first

  1. 2023Grounded Text-to-Image Synthesis with Attention Refocusing8 Jun
  2. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  3. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  5. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  6. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  7. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  8. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  9. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  10. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  11. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts27 Oct
  12. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
8 more
  1. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  2. 2022Blended Latent Diffusion6 Jun
  3. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  4. 2022Pretraining is All You Need for Image-to-Image Translation25 May
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  8. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  11. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
25 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  3. 2019Searching for MobileNetV3cited by 1,984
  4. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  6. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  7. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  8. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  9. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  10. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  11. 2023Zero-shot Image-to-Image Translationcited by 503
  12. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  13. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  14. 2023Universal Guidance for Diffusion Modelscited by 426
  15. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  16. 2022Perception Prioritized Training of Diffusion Modelscited by 301
  17. 2019SEAN: Image Synthesis with Semantic Region-Adaptive Normalizationcited by 253
  18. 2017Photographic Image Synthesis with Cascaded Refinement Networkscited by 223
  19. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  20. 2020You Only Need Adversarial Supervision for Semantic Image Synthesiscited by 129
  21. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  22. 2021Multimodal Conditional Image Synthesis with Product-of-Experts GANscited by 75
  23. 2018NSML: Meet the MLaaS platform with a real-world case studycited by 70
  24. 2023Localized Text-to-Image Generation for Free via Cross Attention Controlcited by 27
  25. 2021On-the-Fly Attention Modulation for Neural Generationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 74; 2025: 51; 2026: 25.

162 papers.

Sort:

By area:

  1. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  2. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  3. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  4. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  5. 2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 142
  6. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 104
  7. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 92
  8. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  9. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 68
  10. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60
  11. 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Modelscited by 54
  12. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Promptscited by 53
38 more
  1. 2025VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editingcited by 50
  2. 2024Region-Aware Text-to-Image Generation via Hard Binding and Soft Refinementcited by 37
  3. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
  4. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  5. 2024Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidancecited by 33
  6. 2025EliGen: Entity-Level Controlled Image Generation with Regional Attentioncited by 32
  7. 2023Fair Text-to-Image Diffusion via Fair Mappingcited by 31
  8. 2024Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Modelscited by 30
  9. 2023R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generationcited by 29
  10. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Modelscited by 27
  11. 2024MC2: Multi-concept Guidance for Customized Multi-concept Generationcited by 26
  12. 2024Text2LiDAR: Text-guided LiDAR Point Cloud Generation via Equirectangular Transformercited by 26
  13. 2024Training-free Regional Prompting for Diffusion Transformerscited by 26
  14. 2024Separable Multi-Concept Erasure from Diffusion Modelscited by 25
  15. 2024NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Mergingcited by 25
  16. 2024SmartControl: Enhancing ControlNet for Handling Rough Visual Conditionscited by 24
  17. 2024GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantationcited by 24
  18. 2025VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generationcited by 22
  19. 2023LoCo: Locally Constrained Training-Free Layout-to-Image Synthesiscited by 21
  20. 2024PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesiscited by 20
  21. 2023Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Modelscited by 19
  22. 2024Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Functioncited by 19
  23. 2025Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformerscited by 19
  24. 2023Fine-grained Controllable Video Generation via Object Appearance and Contextcited by 18
  25. 2023SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Controlcited by 18
  26. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  27. 2024IFAdapter: Instance Feature Control for Grounded Text-to-Image Generationcited by 18
  28. 2024EVA: Zero-shot Accurate Attributes and Multi-Object Video Editingcited by 17
  29. 2024Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjectscited by 16
  30. 2024MagicEraser: Erasing Any Objects via Semantics-Aware Controlcited by 15
  31. 2023PICTURE: PhotorealistIC virtual Try-on from UnconstRained dEsignscited by 14
  32. 2025Object-level Visual Prompts for Compositional Image Generationcited by 14
  33. 2025ComfyGPT: A Self-Optimizing Multi-Agent System for Comprehensive ComfyUI Workflow Generationcited by 14
  34. 2025Towards Transformer-Based Aligned Generation with Self-Coherence Guidancecited by 14
  35. 2023InteractDiffusion: Interaction Control in Text-to-Image Diffusion Modelscited by 13
  36. 2024Customizing Text-to-Image Diffusion with Object Viewpoint Controlcited by 13
  37. 2024PartCraft: Crafting Creative Objects by Partscited by 13
  38. 2025Learning an Image Editing Model without Image Editing Pairscited by 13

50 most cited of 162 All 162 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis
  2. 2023Reason out Your Layout: Evoking the Layout Master from Large Language Models for Text-to-Image Synthesis
  3. 2023Masked-Attention Diffusion Guidance for Spatially Controlling Text-to-Image Generation
  4. 2025Grounding Text-to-Image Diffusion Models for Controlled High-Quality Image Generation
  5. 2023Paragraph-to-Image Generation with Information-Enriched Diffusion Model
  6. 2023Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model
  7. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models
  8. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models