mlsanityData to 7 Oct 2026

Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding

arXiv 2408.08252 ↗First posted 15 Aug 2024cs.LG cs.AI q-bio.GNcited by 127

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models excel at capturing the natural design spaces of images, molecules, DNA, RNA, and protein sequences. However, rather than merely generating designs that are natural, we often aim to optimize downstream reward functions while preserving the naturalness of these design spaces. Existing methods for achieving this goal often require ``differentiable'' proxy models (textite.g., classifier guidance or DPS) or involve computationally expensive fine-tuning of diffusion models (textite.g., classifier-free guidance, RL-based fine-tuning). In our work, we propose a new method to address these challenges. Our algorithm is an iterative sampling method that integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the future, into the standard inference procedure of pre-trained diffusion models. Notably, our approach avoids fine-tuning generative models and eliminates the need to construct differentiable models. This enables us to (1) directly utilize non-differentiable features/reward feedback, commonly used in many scientific domains, and (2) apply our method to recent discrete diffusion models in a principled way. Finally, we demonstrate the effectiveness of our algorithm across several domains, including image generation, molecule generation, and DNA/RNA sequence generation. The code is available at hrefhttps://github.com/masa-ue/SVDDhttps://github.com/masa-ue/SVDD.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 69 of its 107 references that are in the map. First posted on arXiv 15 Aug 2024.

Builds on

69 of 107 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Amortizing intractable inference in diffusion models for vision, language, and control31 May
  2. 2024Gradient Guidance for Diffusion Models: An Optimization Perspective23 Apr
  3. 2024Particle Denoising Diffusion Sampler9 Feb
  4. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion1 Oct
  5. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model17 Mar
  6. 2023Universal Guidance for Diffusion Models14 Feb
  7. 2022Classifier-Free Diffusion Guidance26 Jul
  8. 2022A Continuous Time Framework for Discrete Denoising Models30 May
  9. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
  10. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  11. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  12. 2020Denoising Diffusion Implicit Models6 Oct
1 more
  1. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 56

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2018How Powerful are Graph Neural Networks?cited by 4,355
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  8. 2022Video Diffusion Modelscited by 2,280
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  12. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
38 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  3. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  4. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  5. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  6. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  8. 2018Junction Tree Variational Autoencoder for Molecular Graph Generationcited by 647
  9. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  10. 2022Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Modelcited by 574
  11. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 503
  12. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  13. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  14. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  15. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  16. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 294
  17. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  18. 2019A Theory of Regularized Markov Decision Processescited by 279
  19. 2022Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equationscited by 273
  20. 2017Gate-Variants of Gated Recurrent Unit (GRU) Neural Networkscited by 239
  21. 2022Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problemcited by 229
  22. 2023Practical and Asymptotically Exact Conditional Sampling in Diffusion Modelscited by 218
  23. 2023Geometric Latent Diffusion Models for 3D Molecule Generationcited by 185
  24. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  25. 2023Protein Design with Guided Discrete Diffusioncited by 154
  26. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  27. 2023Controlled Decoding from Language Modelscited by 133
  28. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 124
  29. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  30. 2024Dirichlet Flow Matching with Applications to DNA Sequence Designcited by 110
  31. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  32. 2022Exploring Chemical Space with Score-based Out-of-distribution Generationcited by 95
  33. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  34. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  35. 2023Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programscited by 79
  36. 2023Dirichlet Diffusion Score Model for Biological Sequence Generationcited by 73
  37. 2021Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generationcited by 68
  38. 2023Monte Carlo guided Diffusion for Bayesian linear inverse problemscited by 65

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 75; 2026: 46.

127 papers.

Sort:

By area:

  1. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  2. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 201
  3. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 89
  4. 2025Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Expertscited by 85
  5. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
  6. 2024How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Frameworkcited by 48
  7. 2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 48
  8. 2025Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithmscited by 47
  9. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 40
  10. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 38
  11. 2025Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcingcited by 37
  12. 2026Categorical Flow Mapscited by 37
38 more
  1. 2025Dynamic Search for Inference-Time Alignment in Diffusion Modelscited by 36
  2. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  3. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 32
  4. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 31
  5. 2025Debiasing Guidance for Discrete Diffusion with Sequential Monte Carlocited by 29
  6. 2024Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Predictioncited by 28
  7. 2025Training-Free Guidance Beyond Differentiability: Scalable Path Steering with Tree Search in Diffusion and Flow Modelscited by 23
  8. 2025Reward-Guided Iterative Refinement in Diffusion Models at Test-Time with Applications to Protein and DNA Designcited by 21
  9. 2025Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlocited by 18
  10. 2025Split Gibbs Discrete Diffusion Posterior Samplingcited by 18
  11. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  12. 2025Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Modelscited by 15
  13. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 15
  14. 2025Lookahead Unmasking Elicits Accurate Decoding in Diffusion Language Modelscited by 15
  15. 2025CoDe: Blockwise Control for Denoising Diffusion Modelscited by 14
  16. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
  17. 2025SteerX: Creating Any Camera-Free 3D and 4D Scenes with Geometric Steeringcited by 13
  18. 2025TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusioncited by 13
  19. 2025Inference-Time Scaling of Discrete Diffusion Models via Importance Weighting and Optimal Proposal Designcited by 12
  20. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
  21. 2025Fast and Accurate Causal Parallel Decoding using Jacobi Forcingcited by 11
  22. 2025Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Designcited by 10
  23. 2026Discrete Adjoint Matchingcited by 10
  24. 2024Plug-and-Play Controllable Generation for Discrete Masked Modelscited by 9
  25. 2025CREPE: Controlling Diffusion with Replica Exchangecited by 9
  26. 2025ImagerySearch: Adaptive Test-Time Search for Video Generation Beyond Semantic Dependency Constraintscited by 9
  27. 2025Fast Direct: Query-Efficient Online Black-box Guidance for Diffusion-model Target Generationcited by 7
  28. 2025Adaptive Inference-Time Scaling via Cyclic Diffusion Searchcited by 7
  29. 2025Steering Generative Models with Experimental Data for Protein Fitness Optimizationcited by 7
  30. 2025Inference-Time Scaling of Diffusion Language Models via Trajectory Refinementcited by 7
  31. 2025Effective Test-Time Scaling of Discrete Diffusion through Iterative Refinementcited by 7
  32. 2025Zero-Shot Trajectory Planning for Signal Temporal Logic Taskscited by 6
  33. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Modelscited by 6
  34. 2025Projected Coupled Diffusion for Test-Time Constrained Joint Generationcited by 6
  35. 2025Feynman-Kac-Flow: Inference Steering of Conditional Flow Matching to an Energy-Tilted Posteriorcited by 6
  36. 2025Discrete Guidance Matching: Exact Guidance for Discrete Flow Matchingcited by 6
  37. 2026Training-Free Adaptation of Diffusion Models via Doob's h-Transformcited by 6
  38. 2026Proprio: Latent Self-Scoring and Inference-Time Refinement for Physically Plausible Video Generationcited by 6

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Simple Guidance Mechanisms for Discrete Diffusion Models
  2. 2025Learning Diffusion Models with Flexible Representation Guidance
  3. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design
  4. 2023Bridging the Gap: Addressing Discrepancies in Diffusion Model Training for Classifier-Free Guidance
  5. 2023Towards Controllable Diffusion Models via Reward-Guided Exploration
  6. 2025Towards a unified framework for guided diffusion models
  7. 2024Feedback Efficient Online Fine-Tuning of Diffusion Models
  8. 2024TFG: Unified Training-Free Guidance for Diffusion Models