mlsanityData to 7 Oct 2026

Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models

arXiv 2402.05375 ↗First posted 8 Feb 2024cs.CVcited by 62

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

The success of recent text-to-image diffusion models is largely due to their capacity to be guided by a complex text prompt, which enables users to precisely describe the desired content. However, these models struggle to effectively suppress the generation of undesired content, which is explicitly requested to be omitted from the generated image in the prompt. In this paper, we analyze how to manipulate the text embeddings and remove unwanted content from them. We introduce two contributions, which we refer to as textitsoft-weighted regularization and textitinference-time text embedding optimization. The first regularizes the text embedding matrix and effectively suppresses the undesired content. The second method aims to further suppress the unwanted content generation of the prompt, and encourages the generation of desired content. We evaluate our method quantitatively and qualitatively on extensive experiments, validating its effectiveness. Furthermore, our method is generalizability to both the pixel-space diffusion models (i.e. DeepFloyd-IF) and the latent-space diffusion models (i.e. Stable Diffusion).

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 31 of its 38 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

31 of 38 references.

Closest ancestors · 17

Same direction, newest first

  1. 2023Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing27 Sep
  2. 2023Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Models26 May
  3. 2023StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editing28 Mar
  4. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Models20 Mar
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  6. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  7. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  8. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
  9. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  10. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  11. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
5 more
  1. 2022Blended Latent Diffusion6 Jun
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  5. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov

Other references · 14

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  8. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  9. 2021Grounded Language-Image Pre-trainingcited by 1,147
  10. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  11. 2023Erasing Concepts from Diffusion Modelscited by 530
  12. 2023Zero-shot Image-to-Image Translationcited by 503
2 more
  1. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  2. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 297

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 20; 2026: 22.

62 papers.

Sort:

By area:

  1. 2025SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoderscited by 75
  2. 2024Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidancecited by 63
  3. 2024Controlling Language and Diffusion Models by Transporting Activationscited by 59
  4. 2024R.A.C.E.: Robust Adversarial Concept Erasure for Secure Text-to-Image Diffusion Modelcited by 56
  5. 2024UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Modelscited by 47
  6. 2024Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directionscited by 39
  7. 2024Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matterscited by 38
  8. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  9. 2024Dynamic Negative Guidance of Diffusion Modelscited by 28
  10. 2024Fantastic Copyrighted Beasts and How (Not) to Generate Themcited by 23
  11. 2024Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Functioncited by 19
  12. 2024Targeted Unlearning with Single Layer Unlearning Gradientcited by 16
38 more
  1. 2025TRCE: Towards Reliable Malicious Concept Erasure in Text-to-Image Diffusion Modelscited by 14
  2. 2024AudioEditor: A Training-Free Diffusion-Based Audio Editing Frameworkcited by 13
  3. 2025Fine-Tuning Visual Autoregressive Models for Subject-Driven Generationcited by 13
  4. 2024Backdooring Bias (B2) into Stable Diffusion Modelscited by 12
  5. 2024Compositional Video Generation as Flow Equalizationcited by 12
  6. 2024Replication in Visual Diffusion Models: A Survey and Outlookcited by 11
  7. 2024Growth Inhibitors for Suppressing Inappropriate Image Concepts in Diffusion Modelscited by 10
  8. 2025A Comprehensive Survey on Concept Erasure in Text-to-Image Diffusion Modelscited by 9
  9. 2025Rethinking Machine Unlearning in Image Generation Modelscited by 9
  10. 2025Free-Lunch Color-Texture Disentanglement for Stylized Image Generationcited by 8
  11. 2025SAEmnesia: Erasing Concepts in Diffusion Models with Supervised Sparse Autoencoderscited by 8
  12. 2025EraseFlow: Learning Concept Erasure Policies via GFlowNet-Driven Alignmentcited by 8
  13. 2024Model Integrity when Unlearning with T2I Diffusion Modelscited by 7
  14. 2025CE-SDWV: Effective and Efficient Concept Erasure for Text-to-Image Diffusion Models via a Semantic-Driven Word Vocabularycited by 7
  15. 2025Dynamic Attention Analysis for Backdoor Detection in Text-to-Image Diffusion Modelscited by 3
  16. 2025From Cradle to Cane: A Two-Pass Framework for High-Fidelity Lifespan Face Agingcited by 3
  17. 2025AgeBooth: Controllable Facial Aging and Rejuvenation via Diffusion Modelscited by 3
  18. 2026AEGIS: Adversarial Target-Guided Retention-Data-Free Robust Concept Erasure from Diffusion Modelscited by 3
  19. 2026CleanStyle: Plug-and-Play Style Conditioning Purification for Text-to-Image Stylizationcited by 3
  20. 2025GrOCE:Graph-Guided Online Concept Erasure for Text-to-Image Diffusion Modelscited by 2
  21. 2026Attention, May I Have Your Decision? Localizing Generative Choices in Diffusion Modelscited by 2
  22. 2026Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speechcited by 2
  23. 2024DECOR:Decomposition and Projection of Text Embeddings for Text-to-Image Customizationcited by 1
  24. 2025Safer Prompts: Reducing Risks from Memorization in Visual Generative AIcited by 1
  25. 2025Do Concept Replacement Techniques Really Erase Unacceptable Concepts?cited by 1
  26. 2025Translation of Text Embedding via Delta Vector to Suppress Strongly Entangled Content in Text-to-Image Diffusion Modelscited by 1
  27. 2025Which Way from B to A: The role of embedding geometry in image interpolation for Stable Diffusioncited by 1
  28. 2025Towards Irreversible Machine Unlearning for Diffusion Modelscited by 1
  29. 2026Critic-Guided Reinforcement Unlearning in Text-to-Image Diffusioncited by 1
  30. 2026SPARE: Self-distillation for PARameter-Efficient Removalcited by 1
  31. 2026Prototype-Guided Concept Erasure in Diffusion Modelscited by 1
  32. 2026Rethinking Structure Preservation in Text-Guided Image Editing with Visual Autoregressive Modelscited by 1
  33. 2026Disentangled Sparse Representations for Concept-Separated Diffusion Unlearningcited by 1
  34. 2024Understanding the Limitations of Diffusion Concept Algebra Through Foodcited by 0
  35. 2024FameBias: Embedding Manipulation Bias Attack in Text-to-Image Modelscited by 0
  36. 2025PiCo: Enhancing Text-Image Alignment with Improved Noise Selection and Precise Mask Control in Diffusion Modelscited by 0
  37. 2025ASemConsist: Adaptive Semantic Feature Control for Training-Free Identity-Consistent Generationcited by 0
  38. 2026StoryTailor:A Zero-Shot Pipeline for Action-Rich Multi-Subject Visual Narrativescited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024On Discrete Prompt Optimization for Diffusion Models
  2. 2026ImageRAGTurbo: Towards One-step Text-to-Image Generation with Retrieval-Augmented Diffusion Models
  3. 2025GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Models
  4. 2025Safe and Reliable Diffusion Models via Subspace Projection
  5. 2025Reusing Computation in Text-to-Image Diffusion for Efficient Generation of Image Sets
  6. 2025Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models
  7. 2024A Cat Is A Cat (Not A Dog!): Unraveling Information Mix-ups in Text-to-Image Encoders through Causal Analysis and Embedding Optimization
  8. 2025TPSO: Training-Free Diverse Image Generation via Semantic Prompt Embedding Optimization