# Dynamic Negative Guidance of Diffusion Models

arXiv: https://arxiv.org/abs/2410.14398 · HTML: https://mlsanity.com/paper/2410.14398/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 40 of its 50 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 40)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)
- [Your Diffusion Model is Secretly a Zero-Shot Classifier](https://mlsanity.com/paper/2303.16203/) (2023)
- [Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.17591/) (2023)
- [FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model](https://mlsanity.com/paper/2303.09833/) (2023)
- [Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models](https://mlsanity.com/paper/2404.07724/) (2024)
- [Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC](https://mlsanity.com/paper/2302.11552/) (2023)
- [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://mlsanity.com/paper/1706.08500/) (2017)
- [Selective Amnesia: A Continual Learning Approach to Forgetting in Deep Generative Models](https://mlsanity.com/paper/2305.10120/) (2023)
- [Dynamical Regimes of Diffusion Models](https://mlsanity.com/paper/2402.18491/) (2024)
- [Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyond](https://mlsanity.com/paper/2304.04968/) (2023)
- [VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena](https://mlsanity.com/paper/2112.07566/) (2021)
- [A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data](https://mlsanity.com/paper/2402.16991/) (2024)
- [Circumventing Concept Erasure Methods For Text-to-Image Generative Models](https://mlsanity.com/paper/2308.01508/) (2023)
- [Spontaneous Symmetry Breaking in Generative Diffusion Models](https://mlsanity.com/paper/2305.19693/) (2023)
- [Analysis of Classifier-Free Guidance Weight Schedulers](https://mlsanity.com/paper/2404.13040/) (2024)

## Built on by (most cited first, 28 of 28)

- [Contrastive Flow Matching](https://mlsanity.com/paper/2506.05350/) (2025)
- [Training-Free Safe Denoisers for Safe Use of Diffusion Models](https://mlsanity.com/paper/2502.08011/) (2025)
- [Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlo](https://mlsanity.com/paper/2502.12786/) (2025)
- [Visual Generation Without Guidance](https://mlsanity.com/paper/2501.15420/) (2025)
- [Feedback Guidance of Diffusion Models](https://mlsanity.com/paper/2506.06085/) (2025)
- [Learn to Guide Your Diffusion Model](https://mlsanity.com/paper/2510.00815/) (2025)
- [When Safety Collides: Resolving Multi-Category Harmful Conflicts in Text-to-Image Diffusion via Adaptive Safety Guidance](https://mlsanity.com/paper/2602.20880/) (2026)
- [A Comprehensive Survey on Concept Erasure in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2502.14896/) (2025)
- [The Entropic Signature of Class Speciation in Diffusion Models](https://mlsanity.com/paper/2602.09651/) (2026)
- [Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks](https://mlsanity.com/paper/2607.07907/) (2026)
- [Measuring Semantic Information Production in Generative Diffusion Models](https://mlsanity.com/paper/2506.10433/) (2025)
- [Breaking Lock-In: Preserving Steerability under Low-Data VLA Post-Training](https://mlsanity.com/paper/2604.23121/) (2026)
- [VSF: Simple, Efficient, and Effective Negative Guidance in Few-Step Image Generation Models By Value Sign Flip](https://mlsanity.com/paper/2508.10931/) (2025)
- [Selective Classifier-free Guidance for Zero-shot Text-to-speech](https://mlsanity.com/paper/2509.19668/) (2025)
- [Emergence of Distortions in High-Dimensional Guided Diffusion Models](https://mlsanity.com/paper/2602.00716/) (2026)
- [Synthetic Data Augmentation using Pre-trained Diffusion Models for Long-tailed Food Image Classification](https://mlsanity.com/paper/2506.01368/) (2025)
- [NegFaceDiff: The Power of Negative Context in Identity-Conditioned Diffusion for Synthetic Face Generation](https://mlsanity.com/paper/2508.09661/) (2025)
- [Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models](https://mlsanity.com/paper/2605.08434/) (2026)
- [SafeCFG: Controlling Harmful Features with Dynamic Safe Guidance for Safe Generation](https://mlsanity.com/paper/2412.16039/) (2024)
- [Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearning](https://mlsanity.com/paper/2510.17917/) (2025)
- [EGLOCE: Training-Free Energy-Guided Latent Optimization for Concept Erasure](https://mlsanity.com/paper/2604.09405/) (2026)
- [ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts](https://mlsanity.com/paper/2411.17077/) (2024)
- [Dynamic VLM-Guided Negative Prompting for Diffusion Models](https://mlsanity.com/paper/2510.26052/) (2025)
- [Semantic Anchoring for Robust Personalization in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2511.22245/) (2025)
- [Concept Removal Guidance: Evidence-Calibrated Negative Guidance for Safe Diffusion Sampling](https://mlsanity.com/paper/2606.29801/) (2026)
- [Optimal Self-Distillation for Rectified Flow via Linear Probing](https://mlsanity.com/paper/2607.14947/) (2026)
- [Signed Rectified Flow: Negativity-Controlled Generation](https://mlsanity.com/paper/2607.18516/) (2026)
- [Specificity-Aware Diffusion Steering via Variance-Reduced Sequential Monte Carlo](https://mlsanity.com/paper/2610.00395/) (2026)

## Abstract (arXiv)

Negative Prompting (NP) is widely utilized in diffusion models, particularly in text-to-image applications, to prevent the generation of undesired features. In this paper, we show that conventional NP is limited by the assumption of a constant guidance scale, which may lead to highly suboptimal results, or even complete failure, due to the non-stationarity and state-dependence of the reverse process. Based on this analysis, we derive a principled technique called Dynamic Negative Guidance, which relies on a near-optimal time and state dependent modulation of the guidance without requiring additional training. Unlike NP, negative guidance requires estimating the posterior class probability during the denoising process, which is achieved with limited additional computational overhead by tracking the discrete Markov Chain during the generative process. We evaluate the performance of DNG class-removal on MNIST and CIFAR10, where we show that DNG leads to higher safety, preservation of class balance and image quality when compared with baseline methods. Furthermore, we show that it is possible to use DNG with Stable Diffusion to obtain more accurate and less invasive guidance than NP.