# Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance

arXiv: https://arxiv.org/abs/2404.05384 · HTML: https://mlsanity.com/paper/2404.05384/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 50 of its 55 references that are in the map.
- First posted on arXiv 8 Apr 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)

## Built on by (most cited first, 30 of 59)

- [An analytic theory of creativity in convolutional diffusion models](https://mlsanity.com/paper/2412.20292/) (2024)
- [DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion](https://mlsanity.com/paper/2407.12899/) (2024)
- [Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models](https://mlsanity.com/paper/2406.11831/) (2024)
- [Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models](https://mlsanity.com/paper/2505.11245/) (2025)
- [ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generation](https://mlsanity.com/paper/2511.11483/) (2025)
- [AnyStory: Towards Unified Single and Multiple Subject Personalization in Text-to-Image Generation](https://mlsanity.com/paper/2501.09503/) (2025)
- [Feedback Guidance of Diffusion Models](https://mlsanity.com/paper/2506.06085/) (2025)
- [Learn to Guide Your Diffusion Model](https://mlsanity.com/paper/2510.00815/) (2025)
- [Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Maps](https://mlsanity.com/paper/2411.15236/) (2024)
- [Navigating with Annealing Guidance Scale in Diffusion Space](https://mlsanity.com/paper/2506.24108/) (2025)
- [Adaptive Diffusion Guidance via Stochastic Optimal Control](https://mlsanity.com/paper/2505.19367/) (2025)
- [C2FG: Control Classifier-Free Guidance via Score Discrepancy Analysis](https://mlsanity.com/paper/2603.08155/) (2026)
- [LuciBot: Automated Robot Policy Learning from Generated Videos](https://mlsanity.com/paper/2503.09871/) (2025)
- [Emergence of Distortions in High-Dimensional Guided Diffusion Models](https://mlsanity.com/paper/2602.00716/) (2026)
- [DyWeight: Dynamic Gradient Weighting for Few-Step Diffusion Sampling](https://mlsanity.com/paper/2603.11607/) (2026)
- [AccelAes: Accelerating Diffusion Transformers for Training-Free Aesthetic-Enhanced Image Generation](https://mlsanity.com/paper/2603.12575/) (2026)
- [Delta Score Matters! Spatial Adaptive Multi Guidance in Diffusion Models](https://mlsanity.com/paper/2604.26503/) (2026)
- [APT: Adaptive Personalized Training for Diffusion Models with Limited Data](https://mlsanity.com/paper/2507.02687/) (2025)
- [Asynchronous Denoising Diffusion Models for Aligning Text-to-Image Generation](https://mlsanity.com/paper/2510.04504/) (2025)
- [Making Training-Free Diffusion Segmentors Scale with the Generative Power](https://mlsanity.com/paper/2603.06178/) (2026)
- [To Erase, or Not to Erase: Robust Training-Free Concept Erasure with Preservation aware Adaptive Ranked Subspace Expansion](https://mlsanity.com/paper/2607.23492/) (2026)
- [Pretrained Reversible Generation as Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2412.01787/) (2024)
- [LOCATEdit: Graph Laplacian Optimized Cross Attention for Localized Text-Guided Image Editing](https://mlsanity.com/paper/2503.21541/) (2025)
- [Inverse Design of Metamaterials with Manufacturing-Guiding Spectrum-to-Structure Conditional Diffusion Model](https://mlsanity.com/paper/2506.07083/) (2025)
- [SoftCFG: Uncertainty-guided Stable Guidance for Visual Autoregressive Model](https://mlsanity.com/paper/2510.00996/) (2025)
- [Controllable Stylistic Text Generation with Train-Time Attribute-Regularized Diffusion](https://mlsanity.com/paper/2510.06386/) (2025)
- [PersonaCraft: Personalized and Controllable Full-Body Multi-Human Scene Generation Using Occlusion-Aware 3D-Conditioned Diffusion](https://mlsanity.com/paper/2411.18068/) (2024)
- [RFMI: Estimating Mutual Information on Rectified Flow for Text-to-Image Alignment](https://mlsanity.com/paper/2503.14358/) (2025)
- [Self-NPO: Data-Free Diffusion Model Enhancement via Truncated Diffusion Fine-Tuning](https://mlsanity.com/paper/2505.11777/) (2025)
- [Brownian Bridge Diffusion for Sequential Recommendation](https://mlsanity.com/paper/2507.06121/) (2025)

## Abstract (arXiv)

Classifier-Free Guidance (CFG) has been widely used in text-to-image diffusion models, where the CFG scale is introduced to control the strength of text guidance on the whole image space. However, we argue that a global CFG scale results in spatial inconsistency on varying semantic strengths and suboptimal image quality. To address this problem, we present a novel approach, Semantic-aware Classifier-Free Guidance (S-CFG), to customize the guidance degrees for different semantic units in text-to-image diffusion models. Specifically, we first design a training-free semantic segmentation method to partition the latent image into relatively independent semantic regions at each denoising step. In particular, the cross-attention map in the denoising U-net backbone is renormalized for assigning each patch to the corresponding token, while the self-attention map is used to complete the semantic regions. Then, to balance the amplification of diverse semantic units, we adaptively adjust the CFG scales across different semantic regions to rescale the text guidance degrees into a uniform level. Finally, extensive experiments demonstrate the superiority of S-CFG over the original CFG strategy on various text-to-image diffusion models, without requiring any extra training cost. our codes are available at https://github.com/SmilesDZgk/S-CFG.