# Understanding and Improving Training-free Loss-based Diffusion Guidance

arXiv: https://arxiv.org/abs/2403.12404 · HTML: https://mlsanity.com/paper/2403.12404/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 44 of its 54 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)
- Deep reinforcement learning methods / [RL with diffusion and flow models](https://mlsanity.com/deeprl/rl-with-diffusion-and-flow-models/)

## Builds on (most cited first, 30 of 44)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [Equivariant Diffusion for Molecule Generation in 3D](https://mlsanity.com/paper/2203.17003/) (2022)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition](https://mlsanity.com/paper/1608.04636/) (2016)
- [Improving Diffusion Models for Inverse Problems using Manifold Constraints](https://mlsanity.com/paper/2206.00941/) (2022)
- [Universal Guidance for Diffusion Models](https://mlsanity.com/paper/2302.07121/) (2023)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [Differentiable Augmentation for Data-Efficient GAN Training](https://mlsanity.com/paper/2006.10738/) (2020)
- [Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers](https://mlsanity.com/paper/1906.04584/) (2019)
- [Do Adversarially Robust ImageNet Models Transfer Better?](https://mlsanity.com/paper/2007.08489/) (2020)
- [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023)
- [FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model](https://mlsanity.com/paper/2303.09833/) (2023)
- [Adversarial Attacks and Defences: A Survey](https://mlsanity.com/paper/1810.00069/) (2018)
- [More Control for Free! Image Synthesis with Semantic Diffusion Guidance](https://mlsanity.com/paper/2112.05744/) (2021)

## Built on by (most cited first, 30 of 32)

- [Inference-time Scaling of Diffusion Models through Classical Search](https://mlsanity.com/paper/2505.23614/) (2025)
- [Dynamic Negative Guidance of Diffusion Models](https://mlsanity.com/paper/2410.14398/) (2024)
- [DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Scheduling](https://mlsanity.com/paper/2412.00759/) (2024)
- [Training-Free Guidance Beyond Differentiability: Scalable Path Steering with Tree Search in Diffusion and Flow Models](https://mlsanity.com/paper/2502.11420/) (2025)
- [Feedback Guidance of Diffusion Models](https://mlsanity.com/paper/2506.06085/) (2025)
- [SPARKE: Scalable Prompt-Aware Diversity and Novelty Guidance in Diffusion Models via RKE Score](https://mlsanity.com/paper/2506.10173/) (2025)
- [Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Models](https://mlsanity.com/paper/2506.07177/) (2025)
- [Neuro-Symbolic Generative Diffusion Models for Physically Grounded, Robust, and Safe Generation](https://mlsanity.com/paper/2506.01121/) (2025)
- [Training-Free Guidance for Discrete Diffusion Models for Molecular Generation](https://mlsanity.com/paper/2409.07359/) (2024)
- [RA-DP: Rapid Adaptive Diffusion Policy for Training-Free High-frequency Robotics Replanning](https://mlsanity.com/paper/2503.04051/) (2025)
- [NatADiff: Adversarial Boundary Guidance for Natural Adversarial Diffusion](https://mlsanity.com/paper/2505.20934/) (2025)
- [Flow Matching Posterior Sampling: A Training-free Conditional Generation for Flow Matching](https://mlsanity.com/paper/2411.07625/) (2024)
- [Diffusion Models as Dataset Distillation Priors](https://mlsanity.com/paper/2510.17421/) (2025)
- [Alignment of Diffusion Model and Flow Matching for Text-to-Image Generation](https://mlsanity.com/paper/2602.00413/) (2026)
- [Refining Compositional Diffusion for Reliable Long-Horizon Planning](https://mlsanity.com/paper/2605.03075/) (2026)
- [Boosting Cross-problem Generalization in Diffusion-Based Neural Combinatorial Solver via Inference Time Adaptation](https://mlsanity.com/paper/2502.12188/) (2025)
- [Stein Diffusion Guidance: Training-Free Posterior Correction for Sampling Beyond High-Density Regions](https://mlsanity.com/paper/2507.05482/) (2025)
- [CAPE: Context-Aware Diffusion Policy Via Proximal Mode Expansion for Collision Avoidance](https://mlsanity.com/paper/2511.22773/) (2025)
- [Measurement-Consistent Langevin Corrector for Stabilizing Latent Diffusion Inverse Problem Solvers](https://mlsanity.com/paper/2601.04791/) (2026)
- [LLM Flow Processes for Text-Conditioned Regression](https://mlsanity.com/paper/2601.06147/) (2026)
- [Constrained Diffusion for Accelerated Structure Relaxation of Inorganic Solids with Point Defects](https://mlsanity.com/paper/2602.19153/) (2026)
- [Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs](https://mlsanity.com/paper/2603.25711/) (2026)
- [Physics-Aware Video Generation via Agentic Planning and Graph-Guided Optimization](https://mlsanity.com/paper/2609.13006/) (2026)
- [Guiding Diffusion with Deep Geometric Moments: Balancing Fidelity and Variation](https://mlsanity.com/paper/2505.12486/) (2025)
- [Similarity-Guided Diffusion for Long-Gap Music Inpainting](https://mlsanity.com/paper/2509.16342/) (2025)
- [Guided Diffusion by Optimized Loss Functions on Relaxed Parameters for Inverse Material Design](https://mlsanity.com/paper/2602.15648/) (2026)
- [NDPP-Grasp: Non-Differentiable Physical Plausibility Constraint-Guided Task-Oriented Dexterous Grasp Generation](https://mlsanity.com/paper/2606.02432/) (2026)
- [Not All Prediction Targets Keep Training-Free Diffusion Guidance on the Manifold](https://mlsanity.com/paper/2607.00647/) (2026)
- [Training-free Controllable Human Motion Generation under Heterogeneous Constraints](https://mlsanity.com/paper/2607.01990/) (2026)
- [Integration Matters: Rollout-Based Training for Constrained Diffusion Models](https://mlsanity.com/paper/2607.14398/) (2026)

## Abstract (arXiv)

Adding additional control to pretrained diffusion models has become an increasingly popular research area, with extensive applications in computer vision, reinforcement learning, and AI for science. Recently, several studies have proposed training-free loss-based guidance by using off-the-shelf networks pretrained on clean images. This approach enables zero-shot conditional generation for universal control formats, which appears to offer a free lunch in diffusion guidance. In this paper, we aim to develop a deeper understanding of training-free guidance, as well as overcome its limitations. We offer a theoretical analysis that supports training-free guidance from the perspective of optimization, distinguishing it from classifier-based (or classifier-free) guidance. To elucidate their drawbacks, we theoretically demonstrate that training-free guidance is more susceptible to adversarial gradients and exhibits slower convergence rates compared to classifier guidance. We then introduce a collection of techniques designed to overcome the limitations, accompanied by theoretical rationale and empirical evidence. Our experiments in image and motion generation confirm the efficacy of these techniques.