mlsanityData to 14 Oct 2026

MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask

arXiv 2309.04399 ↗First posted 8 Sep 2023cs.CVcited by 17

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent advancements in diffusion models have showcased their impressive capacity to generate visually striking images. Nevertheless, ensuring a close match between the generated image and the given prompt remains a persistent challenge. In this work, we identify that a crucial factor leading to the text-image mismatch issue is the inadequate cross-modality relation learning between the prompt and the output image. To better align the prompt and image content, we advance the cross-attention with an adaptive mask, which is conditioned on the attention maps and the prompt embeddings, to dynamically adjust the contribution of each text token to the image features. This mechanism explicitly diminishes the ambiguity in semantic information embedding from the text encoder, leading to a boost of text-to-image consistency in the synthesized images. Our method, termed MaskDiffusion, is training-free and hot-pluggable for popular pre-trained diffusion models. When applied to the latent diffusion models, our MaskDiffusion can significantly improve the text-to-image consistency with negligible computation overhead compared to the original diffusion models.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 77 of its 85 references that are in the map. First posted on arXiv 8 Sep 2023.

Builds on

77 of 85 references.

Closest ancestors · 20

Same direction, newest first

  1. 2023Grounded Text-to-Image Synthesis with Attention Refocusing8 Jun
  2. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models23 May
  3. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis7 Apr
  4. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation30 Mar
  5. 2023Directed Diffusion: Direct Control of Object Placement through Attention Guidance25 Feb
  6. 2023LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation16 Feb
  7. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  9. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  10. 2022The Stable Artist: Steering Semantics in Diffusion Latent Space12 Dec
  11. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  12. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
8 more
  1. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Models23 Nov
  2. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Model15 Nov
  3. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  4. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  5. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  8. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  11. 2022Scalable Diffusion Models with Transformerscited by 6,054
  12. 2022Classifier-Free Diffusion Guidancecited by 5,847
38 more
  1. 2022Generative Adversarial Networkscited by 5,627
  2. 2017Neural Discrete Representation Learningcited by 4,712
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  8. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  9. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  10. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  12. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  13. 2016Generative Adversarial Text to Image Synthesiscited by 735
  14. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  15. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 687
  16. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  17. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  18. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  19. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 553
  20. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 532
  21. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  22. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  23. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 499
  24. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  25. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  26. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 334
  27. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
  28. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  29. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 278
  30. 2019Residual Flows for Invertible Generative Modelingcited by 257
  31. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  32. 2022Diffusion models as plug-and-play priorscited by 231
  33. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  34. 2023Implicit Diffusion Models for Continuous Super-Resolutioncited by 218
  35. 2022Diffusion-GAN: Training GANs with Diffusioncited by 205
  36. 2021A Variational Perspective on Diffusion-Based Generative Models and Score Matchingcited by 183
  37. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  38. 2015Generating Images from Captions with Attentioncited by 136

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 7; 2026: 4.

17 papers.

Sort:

By area:

  1. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 247
  2. 2024Magic-Me: Identity-Specific Video Customized Diffusioncited by 59
  3. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  4. 2024Sora Generates Videos with Stunning Geometrical Consistencycited by 31
  5. 2024Few-shot Defect Image Generation based on Consistency Modelingcited by 8
  6. 2025CE-SDWV: Effective and Efficient Concept Erasure for Text-to-Image Diffusion Models via a Semantic-Driven Word Vocabularycited by 8
  7. 2025Conceptrol: Concept Control of Zero-shot Personalized Image Generationcited by 8
  8. 2023ChatAnything: Facetime Chat with LLM-Enhanced Personascited by 3
  9. 2025X2Video: Adapting Diffusion Models for Multimodal Controllable Neural Video Renderingcited by 3
  10. 2025MuseFace: Text-driven Face Editing via Diffusion-based Mask Generation Approachcited by 2
  11. 2025Hierarchical and Step-Layer-Wise Tuning of Attention Specialty for Multi-Instance Synthesis in Diffusion Transformerscited by 2
  12. 2026Rethinking UMM Visual Generation: Masked Modeling for Efficient Image-Only Pre-trainingcited by 2
5 more
  1. 2025Diffusion Prism: Enhancing Diversity and Morphology Consistency in Mask-to-Image Diffusioncited by 1
  2. 2025Geometry-Editable and Appearance-Preserving Object Compositoncited by 1
  3. 2026Semantic Granularity Navigation in Image Editingcited by 1
  4. 2026[CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregationcited by 0
  5. 2026ELDiff: When Evidential Learning Meets Text-to-Image Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models
  2. 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models
  3. 2023Masked-Attention Diffusion Guidance for Spatially Controlling Text-to-Image Generation
  4. 2026STEDiff: Strengthening Text Embedding for Text-to-Image Alignment in Diffusion Model
  5. 2026Rethinking Memorization Mitigation in Diffusion Models: Reinforcing Text Conditioning
  6. 2023Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Models
  7. 2025Rare Text Semantics Were Always There in Your Diffusion Transformer
  8. 2023DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability