mlsanityData to 14 Oct 2026

Understanding the Impact of Negative Prompts: When and How Do They Take Effect?

arXiv 2406.02965 ↗First posted 5 Jun 2024cs.CVcited by 51

On the map: Diffusion models → Robustness, privacy and adversarial aspects

Abstract (arXiv)

The concept of negative prompts, emerging from conditional generation models like Stable Diffusion, allows users to specify what to exclude from the generated images.%, demonstrating significant practical efficacy. Despite the widespread use of negative prompts, their intrinsic mechanisms remain largely unexplored. This paper presents the first comprehensive study to uncover how and when negative prompts take effect. Our extensive empirical analysis identifies two primary behaviors of negative prompts. Delayed Effect: The impact of negative prompts is observed after positive prompts render corresponding content. Deletion Through Neutralization: Negative prompts delete concepts from the generated image through a mutual cancellation effect in latent space with positive prompts. These insights reveal significant potential real-world applications; for example, we demonstrate that negative prompts can facilitate object inpainting with minimal alterations to the background via a simple adaptive algorithm. We believe our findings will offer valuable insights for the community in capitalizing on the potential of negative prompts.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 26 of its 33 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

26 of 33 references.

References · 26

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  9. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  10. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  11. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  12. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
14 more
  1. 2018Generative Image Inpainting with Contextual Attentioncited by 580
  2. 2018nocaps: novel object captioning at scalecited by 444
  3. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  4. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  5. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 379
  6. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 243
  7. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 236
  8. 2015Sequence to Sequence -- Video to Textcited by 158
  9. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  10. 2023Inst-Inpaint: Instructing to Remove Objects with Diffusion Modelscited by 66
  11. 2023Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Modelscited by 40
  12. 2023SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Modelscited by 30
  13. 2020Understanding Guided Image Captioning Performance across Domainscited by 21
  14. 2023Magicremover: Tuning-free Text-guided Image inpainting with Diffusion Modelscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 23; 2026: 16.

51 papers.

Sort:

By area:

  1. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 133
  2. 2024textitRevelio: Interpreting and leveraging semantic information in diffusion modelscited by 30
  3. 2024Dynamic Negative Guidance of Diffusion Modelscited by 28
  4. 2025CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Modelscited by 27
  5. 2025StyleSSP: Sampling StartPoint Enhancement for Training-free Diffusion-based Method for Style Transfercited by 23
  6. 2024Classifier-Free Guidance inside the Attraction Basin May Cause Memorizationcited by 21
  7. 2025Training-Free Safe Denoisers for Safe Use of Diffusion Modelscited by 21
  8. 2024IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning using Instruct Promptscited by 18
  9. 2024ReNeg: Learning Negative Embedding with Reward Guidancecited by 9
  10. 2025Context-guided Responsible Data Augmentation with Diffusion Modelscited by 9
  11. 2024Supercharged One-step Text-to-Image Diffusion Models with Negative Promptscited by 8
  12. 2025CountGD++: Generalized Prompting for Open-World Countingcited by 8
38 more
  1. 2024Safeguarding Text-to-Image Generation via Inference-Time Prompt-Noise Optimizationcited by 7
  2. 2025Zero-Shot Defense Against Toxic Images via Inherent Multimodal Alignment in LVLMscited by 6
  3. 2025What's Producible May Not Be Reachable: Measuring the Steerability of Generative Modelscited by 6
  4. 2026Breaking Lock-In: Preserving Steerability under Low-Data VLA Post-Trainingcited by 6
  5. 2024SimVS: Simulating World Inconsistencies for Robust View Synthesiscited by 5
  6. 2026Embedding-perturbed Exploration Preference Optimization for Flow Modelscited by 5
  7. 2024Negative Token Merging: Image-based Adversarial Feature Guidancecited by 4
  8. 2025Generative Model Unlearning: A Survey through Target Events, Unlearning Operators, and Evaluation Protocolscited by 4
  9. 2025VSF: Simple, Efficient, and Effective Negative Guidance in Few-Step Image Generation Models By Value Sign Flipcited by 4
  10. 2025Selective Classifier-free Guidance for Zero-shot Text-to-speechcited by 4
  11. 2026Emergence of Distortions in High-Dimensional Guided Diffusion Modelscited by 4
  12. 2025AiRacleX: Automated Detection of Price Oracle Manipulations via LLM-Driven Knowledge Mining and Prompt Generationcited by 3
  13. 2025GenColor: Generative Color-Concept Association in Visual Designcited by 3
  14. 2025Wukong Framework for Not Safe For Work Detection in Text-to-Image systemscited by 3
  15. 2025NegFaceDiff: The Power of Negative Context in Identity-Conditioned Diffusion for Synthetic Face Generationcited by 3
  16. 2025VLM-Guided Adaptive Negative Prompting for Creative Generationcited by 3
  17. 2026IDperturb: Enhancing Variation in Synthetic Face Generation via Angular Perturbationcited by 2
  18. 2026Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preferencecited by 2
  19. 2024Stylecodes: Encoding Stylistic Information For Image Generationcited by 1
  20. 2025The LLM Wears Prada: Analysing Gender Bias and Stereotypes through Online Shopping Datacited by 1
  21. 2025A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Modelscited by 1
  22. 2025Erasing 'Ugly' from the Internet: Propagation of the Beauty Myth in Text-Image Modelscited by 1
  23. 2025Prompt-Based Safety Guidance Is Ineffective for Unlearned Text-to-Image Diffusion Modelscited by 1
  24. 2026Attention to Neural Plagiarism: Diffusion Models Can Plagiarize Your Copyrighted Images!cited by 1
  25. 2026DCMorph: Face Morphing via Dual-Stream Cross-Attention Diffusioncited by 1
  26. 2026Obliviate: Erasing Concepts from Autoregressive Image Generation Modelscited by 1
  27. 2026Position: Natural Language Should Not Fully Replace Formal Languagescited by 1
  28. 2024ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Conceptscited by 0
  29. 2025Harnessing the Power of Training-Free Techniques in Text-to-2D Generation for Text-to-3D Generation via Score Distillation Samplingcited by 0
  30. 2025Dynamic VLM-Guided Negative Prompting for Diffusion Modelscited by 0
  31. 2025Target-Balanced Score Distillationcited by 0
  32. 2025Semantic Anchoring for Robust Personalization in Text-to-Image Diffusion Modelscited by 0
  33. 2026CamLit: Unified Video Diffusion with Explicit Camera and Lighting Controlcited by 0
  34. 2026Motion-Guided Semantic Alignment with Negative Prompts for Zero-Shot Video Action Recognitioncited by 0
  35. 2026Guidance Contrastive Token Credit Assignment for Discrete Policy Optimizationcited by 0
  36. 2026SegTune: Structured and Fine-Grained Control for Song Generationcited by 0
  37. 2026Concept Removal Guidance: Evidence-Calibrated Negative Guidance for Safe Diffusion Samplingcited by 0
  38. 2026AU-Guided Synthetic Video Generation for Micro-Expression Recognitioncited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion Models with Adaptive Negative Sampling Without External Resources
  2. 2025Side Effects of Erasing Concepts from Diffusion Models
  3. 2024TraSCE: Trajectory Steering for Concept Erasure
  4. 2023Exploring the Representation Manifolds of Stable Diffusion Through the Lens of Intrinsic Dimension
  5. 2024Optimizing Negative Prompts for Enhanced Aesthetics and Fidelity in Text-To-Image Generation
  6. 2024Improving image synthesis with diffusion-negative sampling
  7. 2025PromptFlare: Prompt-Generalized Defense via Cross-Attention Decoy in Diffusion-Based Inpainting
  8. 2025Erased or Dormant? Rethinking Concept Erasure Through Reversibility