# Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance

arXiv: https://arxiv.org/abs/2403.17377 · HTML: https://mlsanity.com/paper/2403.17377/

## Key facts

- Cited by 113 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 22 of its 64 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 22 of 22)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model](https://mlsanity.com/paper/2212.00490/) (2022)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023)
- [Noise-Free Score Distillation](https://mlsanity.com/paper/2310.17590/) (2023)
- [Return of Unconditional Generation: A Self-supervised Representation Generation Method](https://mlsanity.com/paper/2312.03701/) (2023)
- [Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion](https://mlsanity.com/paper/2312.00852/) (2023)
- [DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization](https://mlsanity.com/paper/2402.09812/) (2024)
- [Readout Guidance: Learning Control from Diffusion Features](https://mlsanity.com/paper/2312.02150/) (2023)
- [LIME: Localized Image Editing via Attention Regularization in Diffusion Models](https://mlsanity.com/paper/2312.09256/) (2023)

## Built on by (most cited first, 30 of 113)

- [Guiding a Diffusion Model with a Bad Version of Itself](https://mlsanity.com/paper/2406.02507/) (2024)
- [Golden Noise for Diffusion Models: A Learning Framework](https://mlsanity.com/paper/2411.09502/) (2024)
- [Phased Consistency Models](https://mlsanity.com/paper/2405.18407/) (2024)
- [Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention](https://mlsanity.com/paper/2408.00760/) (2024)
- [Diffusion Models and Representation Learning: A Survey](https://mlsanity.com/paper/2407.00783/) (2024)
- [No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models](https://mlsanity.com/paper/2407.02687/) (2024)
- [A Noise is Worth Diffusion Guidance](https://mlsanity.com/paper/2412.03895/) (2024)
- [Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Models](https://mlsanity.com/paper/2508.12880/) (2025)
- [Self-Improving Diffusion Models with Synthetic Data](https://mlsanity.com/paper/2408.16333/) (2024)
- [Spatiotemporal Skip Guidance for Enhanced Video Diffusion Sampling](https://mlsanity.com/paper/2411.18664/) (2024)
- [Emergent Temporal Correspondences from Video Diffusion Transformers](https://mlsanity.com/paper/2506.17220/) (2025)
- [Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers](https://mlsanity.com/paper/2509.18096/) (2025)
- [Self-Guidance: Boosting Flow and Diffusion Generation on Their Own](https://mlsanity.com/paper/2412.05827/) (2024)
- [Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models](https://mlsanity.com/paper/2505.11245/) (2025)
- [Lookahead Unmasking Elicits Accurate Decoding in Diffusion Language Models](https://mlsanity.com/paper/2511.05563/) (2025)
- [TCFG: Tangential Damping Classifier-free Guidance](https://mlsanity.com/paper/2503.18137/) (2025)
- [Token Perturbation Guidance for Diffusion Models](https://mlsanity.com/paper/2506.10036/) (2025)
- [Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models](https://mlsanity.com/paper/2505.21179/) (2025)
- [FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Compute](https://mlsanity.com/paper/2502.20126/) (2025)
- [Entropy Rectifying Guidance for Diffusion and Flow Models](https://mlsanity.com/paper/2504.13987/) (2025)
- [It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Models](https://mlsanity.com/paper/2601.00090/) (2025)
- [CAMEO: Correspondence-Attention Alignment for Multi-View Diffusion Models](https://mlsanity.com/paper/2512.03045/) (2025)
- [TV-3DG: Mastering Text-to-3D Customized Generation with Visual Prompt](https://mlsanity.com/paper/2410.21299/) (2024)
- [Gradient-Free Classifier Guidance for Diffusion Model Sampling](https://mlsanity.com/paper/2411.15393/) (2024)
- [Unconditional Priors Matter! Improving Conditional Generation of Fine-Tuned Diffusion Models](https://mlsanity.com/paper/2503.20240/) (2025)
- [PLADIS: Pushing the Limits of Attention in Diffusion Models at Inference Time by Leveraging Sparsity](https://mlsanity.com/paper/2503.07677/) (2025)
- [Understanding Attention Mechanism in Video Diffusion Models](https://mlsanity.com/paper/2504.12027/) (2025)
- [DFBench: Benchmarking Deepfake Image Detection Capability of Large Multimodal Models](https://mlsanity.com/paper/2506.03007/) (2025)
- [Steering Guidance for Personalized Text-to-Image Diffusion Models](https://mlsanity.com/paper/2508.00319/) (2025)
- [Massive Activations are the Key to Local Detail Synthesis in Diffusion Transformers](https://mlsanity.com/paper/2510.11538/) (2025)

## Abstract (arXiv)

Recent studies have demonstrated that diffusion models are capable of generating high-quality samples, but their quality heavily depends on sampling guidance techniques, such as classifier guidance (CG) and classifier-free guidance (CFG). These techniques are often not applicable in unconditional generation or in various downstream tasks such as image restoration. In this paper, we propose a novel sampling guidance, called Perturbed-Attention Guidance (PAG), which improves diffusion sample quality across both unconditional and conditional settings, achieving this without requiring additional training or the integration of external modules. PAG is designed to progressively enhance the structure of samples throughout the denoising process. It involves generating intermediate samples with degraded structure by substituting selected self-attention maps in diffusion U-Net with an identity matrix, by considering the self-attention mechanisms' ability to capture structural information, and guiding the denoising process away from these degraded samples. In both ADM and Stable Diffusion, PAG surprisingly improves sample quality in conditional and even unconditional scenarios. Moreover, PAG significantly improves the baseline performance in various downstream tasks where existing guidances such as CG or CFG cannot be fully utilized, including ControlNet with empty prompts and image restoration such as inpainting and deblurring.