mlsanityData to 14 Oct 2026

Maximize Your Diffusion: A Study into Reward Maximization and Alignment for Diffusion-based Control

arXiv 2502.12198 ↗First posted 16 Feb 2025cs.LG cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Diffusion-based planning, learning, and control methods present a promising branch of powerful and expressive decision-making solutions. Given the growing interest, such methods have undergone numerous refinements over the past years. However, despite these advancements, existing methods are limited in their investigations regarding general methods for reward maximization within the decision-making process. In this work, we study extensions of fine-tuning approaches for control applications. Specifically, we explore extensions and various design choices for four fine-tuning approaches: reward alignment through reinforcement learning, direct preference optimization, supervised fine-tuning, and cascading diffusion. We optimize their usage to merge these independent efforts into one unified paradigm. We show the utility of such propositions in offline RL settings and demonstrate empirical improvements over a rich array of control tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 47 references that are in the map. First posted on arXiv 16 Feb 2025.

Builds on

41 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  2. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep

Other references · 39

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2022Scalable Diffusion Models with Transformerscited by 6,054
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
27 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  8. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  9. 2021Variational Diffusion Modelscited by 1,153
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  11. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  12. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  13. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  14. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  15. 2021Implicit Behavioral Cloningcited by 452
  16. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  17. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  18. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 288
  19. 2024Diffusion Policy Policy Optimizationcited by 280
  20. 2022A Survey on Generative Diffusion Modelcited by 240
  21. 2023Synthetic Experience Replaycited by 101
  22. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81
  23. 2024Rectified Diffusion: Straightness Is Not Your Need in Rectified Flowcited by 61
  24. 2024DiffuserLite: Towards Real-time Diffusion Planningcited by 42
  25. 2024CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Makingcited by 39
  26. 2023World Models via Policy-Guided Trajectory Diffusioncited by 34
  27. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Towards a unified framework for guided diffusion modelscited by 8
  2. 2025Multi-agent Auto-Bidding with Latent Graph Diffusion Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Diffusion Controller: Framework, Algorithms and Parameterization
  2. 2026CRAFT: Aligning Diffusion Models with Fine-Tuning Is Easier Than You Think
  3. 2026Diffusion Reinforcement Learning via Centered Reward Distillation
  4. 2024Task-Agnostic Pre-training and Task-Guided Fine-tuning for Versatile Diffusion Planner
  5. 2023Diffusion Models for Reinforcement Learning: A Survey
  6. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
  7. 2025Accelerating Diffusion Planners in Offline RL via Reward-Aware Consistency Trajectory Distillation
  8. 2026Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View