mlsanityData to 7 Oct 2026

Gradient Guidance for Diffusion Models: An Optimization Perspective

arXiv 2404.14743 ↗First posted 23 Apr 2024stat.ML cs.LGcited by 59

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models have demonstrated empirical successes in various applications and can be adapted to task-specific needs via guidance. This paper studies a form of gradient guidance for adapting a pre-trained diffusion model towards optimizing user-specified objectives. We establish a mathematical framework for guided diffusion to systematically study its optimization theory and algorithmic design. Our theoretical analysis spots a strong link between guided diffusion models and optimization: gradient-guided diffusion models are essentially sampling solutions to a regularized optimization problem, where the regularization is imposed by the pre-training data. As for guidance design, directly bringing in the gradient of an external objective function as guidance would jeopardize the structure in generated samples. We investigate a modified form of gradient guidance based on a forward prediction loss, which leverages the information in pre-trained score functions and provably preserves the latent structure. We further consider an iteratively fine-tuned version of gradient-guided diffusion where guidance and score network are both updated with newly generated samples. This process mimics a first-order optimization iteration in expectation, for which we proved O(1/K) convergence rate to the global optimum when the objective function is concave. Our code will be released at https://github.com/yukang123/GGDMOptim.git.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 59 of its 72 references that are in the map. First posted on arXiv 23 Apr 2024.

Builds on

59 of 72 references.

Closest ancestors · 29

Same direction, newest first

  1. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization11 Apr
  2. 2024Unveil Conditional Diffusion Models with Classifier-free Guidance: A Sharp Statistical Theory18 Mar
  3. 2023Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint Method19 Dec
  4. 2023Manifold Preserving Guided Diffusion28 Nov
  5. 2023Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Models20 Sep
  6. 2023Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization7 Aug
  7. 2023AdjointDPM: Adjoint Sensitivity Method for Gradient Backpropagation of Diffusion Probabilistic Models20 Jul
  8. 2023The probability flow ODE is provably fast19 May
  9. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidance23 Mar
  10. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model17 Mar
  11. 2023Restoration-Degradation Beyond Linear Diffusions: A Non-Asymptotic Analysis For DDIM-Type Samplers6 Mar
  12. 2023Diffusion Models are Minimax Optimal Distribution Estimators3 Mar
17 more
  1. 2023Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data14 Feb
  2. 2023Universal Guidance for Diffusion Models14 Feb
  3. 2022Convergence of score-based generative modeling for general data distributions26 Sep
  4. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions22 Sep
  5. 2022Diffusion Models: A Comprehensive Survey of Methods and Applications2 Sep
  6. 2022Let us Build Bridges: Understanding and Extending Diffusion Generative Models31 Aug
  7. 2022Convergence of denoising diffusion models under the manifold hypothesis10 Aug
  8. 2022Classifier-Free Diffusion Guidance26 Jul
  9. 2022Diffusion models as plug-and-play priors17 Jun
  10. 2022Convergence for score-based generative modeling with polynomial complexity13 Jun
  11. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  12. 2021Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modeling1 Jun
  13. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  14. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  15. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  16. 2020Denoising Diffusion Implicit Models6 Oct
  17. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  5. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
  6. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  8. 2022Denoising Diffusion Restoration Modelscited by 951
  9. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  10. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  11. 2022Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Modelcited by 574
  12. 2014Convex Optimization: Algorithms and Complexitycited by 569
18 more
  1. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  2. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraintscited by 501
  3. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  4. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  5. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  6. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 381
  7. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  8. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  9. 2023Protein Design with Guided Discrete Diffusioncited by 154
  10. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  11. 2020Generative Modeling with Denoising Auto-Encoders and Langevin Samplingcited by 118
  12. 2024D-Flow: Differentiating through Flows for Controlled Generationcited by 115
  13. 2023Optimizing Diffusion Noise Can Serve As Universal Motion Priorscited by 97
  14. 2024Optimal score estimation via empirical Bayes smoothingcited by 60
  15. 2023Reward-Directed Conditional Diffusion: Provable Distribution Estimation and Reward Improvementcited by 49
  16. 2023Image Restoration by Denoising Diffusion Models with Iteratively Preconditioned Guidancecited by 43
  17. 2024Implicit Diffusion: Efficient Optimization through Stochastic Samplingcited by 22
  18. 2023Diffusion Models in Bioinformatics: A New Wave of Deep Learning Revolution in Actioncited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 30; 2026: 17.

59 papers.

Sort:

By area:

  1. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  2. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
  3. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 72
  4. 2024Rectified Diffusion: Straightness Is Not Your Need in Rectified Flowcited by 59
  5. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 38
  6. 2025Dynamic Search for Inference-Time Alignment in Diffusion Modelscited by 36
  7. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNetscited by 30
  8. 2025Training-Free Guidance Beyond Differentiability: Scalable Path Steering with Tree Search in Diffusion and Flow Modelscited by 23
  9. 2024On Statistical Rates of Conditional Diffusion Transformers: Approximation, Estimation and Minimax Optimalitycited by 21
  10. 2025DriftLite: Lightweight Drift Control for Inference-Time Scaling of Diffusion Modelscited by 21
  11. 2025Solving Inverse Problems via Diffusion-Based Priors: An Approximation-Free Ensemble Sampling Approachcited by 18
  12. 2024Adding Conditional Control to Diffusion Models with Reinforcement Learningcited by 16
38 more
  1. 2024InstructG2I: Synthesizing Images from Multimodal Attributed Graphscited by 14
  2. 2025CoDe: Blockwise Control for Denoising Diffusion Modelscited by 14
  3. 2024Diffusion Models as Network Optimizers: Explorations and Analysiscited by 13
  4. 2025LinEAS: End-to-end Learning of Activation Steering with a Distributional Losscited by 12
  5. 2025Physics-Informed Distillation of Diffusion Models for PDE-Constrained Generationcited by 11
  6. 2025Source-Guided Flow Matchingcited by 11
  7. 2025HardFlow: Hard-Constrained Sampling for Flow-Matching Models via Trajectory Optimizationcited by 10
  8. 2025LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimizationcited by 9
  9. 2025Neuro-Symbolic Generative Diffusion Models for Physically Grounded, Robust, and Safe Generationcited by 9
  10. 2024Gradient-Free Classifier Guidance for Diffusion Model Samplingcited by 8
  11. 2024Nonlinear Assimilation via Score-based Sequential Langevin Samplingcited by 7
  12. 2025Projected Coupled Diffusion for Test-Time Constrained Joint Generationcited by 6
  13. 2025Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Rankingcited by 5
  14. 2025HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perceptioncited by 5
  15. 2024Latent Diffusion Models for Controllable RNA Sequence Generationcited by 4
  16. 2025A First-order Generative Bilevel Optimization Framework for Diffusion Modelscited by 4
  17. 2025Tree-Guided Diffusion Plannercited by 4
  18. 2025SONIC: Spectral Optimization of Noise for Inpainting with Consistencycited by 4
  19. 2026Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Modelscited by 4
  20. 2025Generative Diffusion Receivers: Achieving Pilot-Efficient MIMO-OFDM Communicationscited by 3
  21. 2025SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit 3D Meshescited by 3
  22. 2024Differentially Private Kernel Density Estimationcited by 2
  23. 2025Why Settle for Mid: A Probabilistic Viewpoint to Spatial Relationship Alignment in Text-to-image Modelscited by 2
  24. 2025Controllable 3D Molecular Generation for Structure-Based Drug Design Through Bayesian Flow Networks and Gradient Integrationcited by 2
  25. 2025Diffusion Alignment as Variational Expectation-Maximizationcited by 2
  26. 2026Provably Safe Generative Sampling with Constricting Barrier Functionscited by 2
  27. 2026Inference-Time Attribute Distribution Alignment for Unconditional Diffusioncited by 2
  28. 2026Constrained Diffusion Models with Primal-Dual Inferencecited by 2
  29. 2025Stein Diffusion Guidance: Training-Free Posterior Correction for Sampling Beyond High-Density Regionscited by 1
  30. 2025A Gradient Guided Diffusion Framework for Chance Constrained Programmingcited by 1
  31. 2025CAPE: Context-Aware Diffusion Policy Via Proximal Mode Expansion for Collision Avoidancecited by 1
  32. 2026Multi-Subspace Multi-Modal Modeling for Diffusion Models: Estimation, Convergence and Mixture of Expertscited by 1
  33. 2026LLM Flow Processes for Text-Conditioned Regressioncited by 1
  34. 2026Provably Reliable Classifier Guidance via Cross-Entropy Controlcited by 1
  35. 2026Machine Pareidolia: Protecting Facial Image with Emotional Editingcited by 1
  36. 2025When are Diffusion Priors Helpful in Sparse Reconstruction? A Study with Sparse-view CTcited by 0
  37. 2025Efficient Weighted Sampling via Score-based Generative Modelscited by 0
  38. 2025Contact Wasserstein Geodesics for Non-Conservative Schr"odinger Bridgescited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards a unified framework for guided diffusion models
  2. 2023Bridging the Gap: Addressing Discrepancies in Diffusion Model Training for Classifier-Free Guidance
  3. 2024TFG: Unified Training-Free Guidance for Diffusion Models
  4. 2026Learned End-to-End Guidance Schedules for Diffusion Models
  5. 2025Adaptive Diffusion Guidance via Stochastic Optimal Control
  6. 2025REG: Rectified Gradient Guidance for Conditional Diffusion Models
  7. 2024Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models
  8. 2025Variational Control for Guidance in Diffusion Models