# Mitigating Diffusion Model Hallucinations with Dynamic Guidance

arXiv: https://arxiv.org/abs/2510.05356 · HTML: https://mlsanity.com/paper/2510.05356/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 44 of its 49 references that are in the map.
- First posted on arXiv 6 Oct 2025.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 30 of 44)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation](https://mlsanity.com/paper/2403.04692/) (2024)
- [Guiding a Diffusion Model with a Bad Version of Itself](https://mlsanity.com/paper/2406.02507/) (2024)
- [FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model](https://mlsanity.com/paper/2303.09833/) (2023)
- [Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models](https://mlsanity.com/paper/2404.07724/) (2024)
- [Diffusion models as plug-and-play priors](https://mlsanity.com/paper/2206.09012/) (2022)
- [Understanding Hallucinations in Diffusion Models through Mode Interpolation](https://mlsanity.com/paper/2406.09358/) (2024)
- [Analysis of Classifier-Free Guidance Weight Schedulers](https://mlsanity.com/paper/2404.13040/) (2024)
- [Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach](https://mlsanity.com/paper/2405.15613/) (2024)
- [11K Hands: Gender recognition and biometric identification using a large dataset of hand images](https://mlsanity.com/paper/1711.04322/) (2017)
- [Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance](https://mlsanity.com/paper/2402.08680/) (2024)

## Built on by (most cited first, 5 of 5)

- [Local Intrinsic Dimension Unveils Hallucinations in Diffusion Models](https://mlsanity.com/paper/2605.05026/) (2026)
- [Why DDIM Hallucinates More Than DDPM: A Theoretical Analysis of Reverse Dynamics](https://mlsanity.com/paper/2605.06831/) (2026)
- [VIGIL: Tackling Hallucination Detection in Image Recontextualization](https://mlsanity.com/paper/2602.14633/) (2026)
- [Score-Control for Hallucination Reduction in Diffusion Models](https://mlsanity.com/paper/2606.00377/) (2026)
- [Unified Backbone Refinement for Diffusion Models via Internal-Latent Analysis](https://mlsanity.com/paper/2607.09753/) (2026)

## Abstract (arXiv)

Hallucinations in diffusion models are samples with structural inconsistencies that can emerge due to the excessive smoothing of the learned score function, which in turn leads to interpolations between modes of the data distribution. Since semantic interpolations are often desirable and contribute to sample diversity, we believe that a nuanced and targeted solution is required to address diffusion model hallucinations. In this work, we introduce Dynamic Guidance, which mitigates hallucinations by selectively sharpening the score function only along the pre-determined directions known to cause artifacts, while preserving valid semantic variations. This sharpening can be performed using either pre-determined classes or semantically coherent clusters that form pseudo-classes over the data distribution. The latter allows for a principled extension of Dynamic Guidance to text-to-image generation, where we select modes to correspond to fine-grained contextual differences in textual descriptions. To our knowledge, this is the first approach that addresses hallucinations at generation time rather than through post-hoc filtering. Dynamic Guidance substantially reduces hallucinations on both controlled and natural image datasets, significantly outperforming baselines.