mlsanityData to 7 Oct 2026

Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review

arXiv 2501.09685 ↗First posted 16 Jan 2025cs.AI cs.LG q-bio.QMcited by 82

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

This tutorial provides an in-depth guide on inference-time guidance and alignment methods for optimizing downstream reward functions in diffusion models. While diffusion models are renowned for their generative modeling capabilities, practical applications in fields such as biology often require sample generation that maximizes specific metrics (e.g., stability, affinity in proteins, closeness to target structures). In these scenarios, diffusion models can be adapted not only to generate realistic samples but also to explicitly maximize desired measures at inference time without fine-tuning. This tutorial explores the foundational aspects of such inference-time algorithms. We review these methods from a unified perspective, demonstrating that current techniques -- such as Sequential Monte Carlo (SMC)-based guidance, value-based sampling, and classifier guidance -- aim to approximate soft optimal denoising processes (a.k.a. policies in RL) that combine pre-trained denoising processes with value functions serving as look-ahead functions that predict from intermediate states to terminal rewards. Within this framework, we present several novel algorithms not yet covered in the literature. Furthermore, we discuss (1) fine-tuning methods combined with inference-time techniques, (2) inference-time algorithms based on search algorithms such as Monte Carlo tree search, which have received limited attention in current research, and (3) connections between inference-time algorithms in language models and diffusion models. The code of this tutorial on protein design is available at https://github.com/masa-ue/AlignInversePro

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 62 of its 162 references that are in the map. First posted on arXiv 16 Jan 2025.

Builds on

62 of 162 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Models12 Jan
  2. 2024How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Framework4 Oct
  3. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding15 Aug
  4. 2024Informed Correctors for Discrete Diffusion Models30 Jul
  5. 2024Amortizing intractable inference in diffusion models for vision, language, and control31 May
  6. 2024Towards a mathematical theory for consistency training in diffusion models12 Feb
  7. 2024Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorial12 Feb
  8. 2024Particle Denoising Diffusion Sampler9 Feb
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  10. 2023Consistency Models2 Mar
  11. 2022Score-based Continuous-time Discrete Diffusion Models30 Nov
  12. 2022Understanding Diffusion Models: A Unified Perspective25 Aug
3 more
  1. 2022Classifier-Free Diffusion Guidance26 Jul
  2. 2022Progressive Distillation for Fast Sampling of Diffusion Models1 Feb
  3. 2020Denoising Diffusion Implicit Models6 Oct

Other references · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
  6. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  7. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  8. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  9. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  11. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  12. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
35 more
  1. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 511
  2. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  3. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  4. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  5. 2015Policy Distillationcited by 414
  6. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  7. 2022DiffDock: Diffusion Steps, Twists, and Turns for Molecular Dockingcited by 307
  8. 2024Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Designcited by 294
  9. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  10. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  11. 2023Flow Matching on General Geometriescited by 260
  12. 2022Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problemcited by 229
  13. 2023SE(3) diffusion model with application to protein backbone generationcited by 214
  14. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  15. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  16. 2023Controlled Decoding from Language Modelscited by 133
  17. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 124
  18. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  19. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
  20. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  21. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  22. 2023Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programscited by 79
  23. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  24. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 72
  25. 2023Monte Carlo guided Diffusion for Bayesian linear inverse problemscited by 65
  26. 2024DPLM-2: A Multimodal Diffusion Protein Language Modelcited by 59
  27. 2021Machine Translation Decoding beyond Beam Searchcited by 55
  28. 2023Learning to Generate Better Than Your LLMcited by 49
  29. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  30. 2023Protein Discovery with Discrete Walk-Jump Samplingcited by 38
  31. 2024Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Predictioncited by 28
  32. 2024ProteinBench: A Holistic Evaluation of Protein Foundation Modelscited by 23
  33. 2024Generative Flows on Synthetic Pathway for Drug Designcited by 20
  34. 2024Adding Conditional Control to Diffusion Models with Reinforcement Learningcited by 16
  35. 2020Practical Massively Parallel Monte-Carlo Tree Search Applied to Molecular Designcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 37; 2026: 45.

82 papers.

Sort:

By area:

  1. 2025Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Expertscited by 85
  2. 2025Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithmscited by 47
  3. 2025DynaGuide: Steering Diffusion Polices with Active Dynamic Guidancecited by 39
  4. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  5. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 32
  6. 2026Meta Flow Maps enable scalable reward alignmentcited by 32
  7. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 31
  8. 2025Debiasing Guidance for Discrete Diffusion with Sequential Monte Carlocited by 29
  9. 2026Diamond Maps: Efficient Reward Alignment via Stochastic Flow Mapscited by 26
  10. 2025Training-Free Guidance Beyond Differentiability: Scalable Path Steering with Tree Search in Diffusion and Flow Modelscited by 23
  11. 2026Discrete Flow Mapscited by 23
  12. 2025GLASS Flows: Transition Sampling for Alignment of Flow and Diffusion Modelscited by 17
38 more
  1. 2025RNE: plug-and-play diffusion inference-time control and energy-based trainingcited by 16
  2. 2025Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Modelscited by 15
  3. 2025TTS-VAR: A Test-Time Scaling Framework for Visual Auto-Regressive Generationcited by 15
  4. 2025ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videoscited by 12
  5. 2025Inference-Time Scaling of Discrete Diffusion Models via Importance Weighting and Optimal Proposal Designcited by 12
  6. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
  7. 2025Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Designcited by 10
  8. 2025RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scalingcited by 9
  9. 2025MoE-DP: An MoE-Enhanced Diffusion Policy for Robust Long-Horizon Robotic Manipulation with Skill Decomposition and Failure Recoverycited by 9
  10. 2025It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Modelscited by 9
  11. 2025Adaptive Inference-Time Scaling via Cyclic Diffusion Searchcited by 7
  12. 2025Effective Test-Time Scaling of Discrete Diffusion through Iterative Refinementcited by 7
  13. 2025Projected Coupled Diffusion for Test-Time Constrained Joint Generationcited by 6
  14. 2026Training-Free Adaptation of Diffusion Models via Doob's h-Transformcited by 6
  15. 2025Graph Guided Diffusion: Unified Guidance for Conditional Graph Generationcited by 5
  16. 2025ReGuidance: A Simple Diffusion Wrapper for Boosting Sample Quality on Hard Inverse Problemscited by 5
  17. 2025Controllable Graph Generation with Diffusion Models via Inference-Time Tree Search Guidancecited by 5
  18. 2026Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Modelscited by 5
  19. 2026How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidancecited by 5
  20. 2025DDPS: Discrete Diffusion Posterior Sampling for Paths in Layered Graphscited by 4
  21. 2025AutoRefiner: Improving Autoregressive Video Diffusion Models via Reflective Refinement Over the Stochastic Sampling Pathcited by 4
  22. 2026On the Power of (Approximate) Reward Models for Inference-Time Scalingcited by 4
  23. 2026Self-Rewarding Sequential Monte Carlo for Masked Diffusion Language Modelscited by 4
  24. 2025Modular Diffusion Policy Training: Decoupling and Recombining Guidance and Diffusion for Offline RLcited by 3
  25. 2026Reward-Guided Discrete Diffusion via Clean-Sample Markov Chain for Molecule and Biological Sequence Designcited by 3
  26. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 3
  27. 2026Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inferencecited by 3
  28. 2026Ito maps for any-step SDEscited by 3
  29. 2025Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMscited by 2
  30. 2026Inference-Time Alignment for Diffusion Models via Variationally Stable Doob's Matchingcited by 2
  31. 2026Provably Safe Generative Sampling with Constricting Barrier Functionscited by 2
  32. 2026Inference-Time Attribute Distribution Alignment for Unconditional Diffusioncited by 2
  33. 2026Simple Approximation and Derivative Free Inference-Time Scaling for Diffusion Models via Sequential Monte Carlo on Path Measurescited by 2
  34. 2026SURGE: Approximation and Training Free Particle Filter for Diffusion Surrogatecited by 2
  35. 2026Stitched Value Model for Diffusion Alignmentcited by 2
  36. 2026Are we really tilting? The mechanics of reward guidance in flow and diffusion modelscited by 2
  37. 2026Flash-BoN: Instant Drafts for Inference-Time Scaling in Diffusion Modelscited by 2
  38. 2025Inference-Time Alignment of Diffusion Models via Evolutionary Algorithmscited by 1

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Dynamic Search for Inference-Time Alignment in Diffusion Models
  2. 2025Unified Control for Inference-Time Guidance of Denoising Diffusion Models
  3. 2025Test-time Alignment of Diffusion Models without Reward Over-optimization
  4. 2025Reward-Guided Iterative Refinement in Diffusion Models at Test-Time with Applications to Protein and DNA Design
  5. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimization
  6. 2025CoDe: Blockwise Control for Denoising Diffusion Models
  7. 2025DriftLite: Lightweight Drift Control for Inference-Time Scaling of Diffusion Models
  8. 2026A Mean-Field Framework for Inference-Time Distributional Control of Diffusion Models