mlsanityData to 7 Oct 2026

Enhancing Prompt Following with Visual Control Through Training-Free Mask-Guided Diffusion

arXiv 2404.14768 ↗First posted 23 Apr 2024cs.CVcited by 6

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recently, integrating visual controls into text-to-image~(T2I) models, such as ControlNet method, has received significant attention for finer control capabilities. While various training-free methods make efforts to enhance prompt following in T2I models, the issue with visual control is still rarely studied, especially in the scenario that visual controls are misaligned with text prompts. In this paper, we address the challenge of ``Prompt Following With Visual Control" and propose a training-free approach named Mask-guided Prompt Following (MGPF). Object masks are introduced to distinct aligned and misaligned parts of visual controls and prompts. Meanwhile, a network, dubbed as Masked ControlNet, is designed to utilize these object masks for object generation in the misaligned visual control region. Further, to improve attribute matching, a simple yet efficient loss is designed to align the attention maps of attributes with object regions constrained by ControlNet and object masks. The efficacy and superiority of MGPF are validated through comprehensive quantitative and qualitative experiments.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 49 of its 50 references that are in the map. First posted on arXiv 23 Apr 2024.

Builds on

49 of 50 references.

Closest ancestors · 18

Same direction, newest first

  1. 2023Divide & Bind Your Attention for Improved Generative Semantic Nursing20 Jul
  2. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion20 Jul
  3. 2023A-STAR: Test-time Attention Segregation and Retention for Text-to-image Synthesis26 Jun
  4. 2023Energy-Based Cross Attention for Bayesian Context Update in Text-to-Image Diffusion Models16 Jun
  5. 2023Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment15 Jun
  6. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models23 May
  7. 2023Directed Diffusion: Direct Control of Object Placement through Attention Guidance25 Feb
  8. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  10. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  11. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  12. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
6 more
  1. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attention10 Oct
  2. 2022Blended Latent Diffusion6 Jun
  3. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
19 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  2. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  3. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  4. 2022Human Motion Diffusion Modelcited by 1,062
  5. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  6. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  7. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  8. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  9. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  10. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  11. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  12. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  13. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  14. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  15. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  16. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 189
  17. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
  18. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  19. 2023Single Motion Diffusioncited by 65

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 2.

6 papers.

  1. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  2. 2025SUB: Benchmarking CBM Generalization via Synthetic Attribute Substitutionscited by 6
  3. 2024Prompt-Softbox-Prompt: A Free-Text Embedding Control for Image Editingcited by 4
  4. 2024E-Commerce Inpainting with Mask Guidance in Controlnet for Reducing Overcompletioncited by 4
  5. 2026Design Your Ad: Personalized Advertising Image and Text Generation with Unified Autoregressive Modelscited by 1
  6. 2026COLLAR: Cascaded Object-Level Latent Refinement for High-Fidelity Conditional Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mask-ControlNet: Higher-Quality Image Generation with An Additional Mask Prompt
  2. 2023Local Conditional Controlling for Text-to-Image Diffusion Models
  3. 2024Layout-to-Image Generation with Localized Descriptions using ControlNet with Cross-Attention Control
  4. 2023Masked-Attention Diffusion Guidance for Spatially Controlling Text-to-Image Generation
  5. 2024Layout Control and Semantic Guidance with Attention Loss Backward for T2I Diffusion Model
  6. 2025Detail++: Training-Free Detail Enhancer for T2I Diffusion Models
  7. 2025SemanticControl: A Training-Free Approach for Handling Loosely Aligned Visual Conditions in ControlNet
  8. 2023Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Models