mlsanityData to 13 Oct 2026

Forward KL Regularized Preference Optimization for Aligning Diffusion Policies

arXiv 2409.05622 ↗First posted 9 Sep 2024cs.LGcited by 10

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Diffusion models have achieved remarkable success in sequential decision-making by leveraging the highly expressive model capabilities in policy learning. A central problem for learning diffusion policies is to align the policy output with human intents in various tasks. To achieve this, previous methods conduct return-conditioned policy generation or Reinforcement Learning (RL)-based policy optimization, while they both rely on pre-defined reward functions. In this work, we propose a novel framework, Forward KL regularized Preference optimization for aligning Diffusion policies, to align the diffusion policy with preferences directly. We first train a diffusion policy from the offline dataset without considering the preference, and then align the policy to the preference data via direct preference optimization. During the alignment phase, we formulate direct preference learning in a diffusion policy, where the forward KL regularization is employed in preference optimization to avoid generating out-of-distribution actions. We conduct extensive experiments for MetaWorld manipulation and D4RL tasks. The results show our method exhibits superior alignment with preferences and outperforms previous state-of-the-art algorithms.

Key facts. Cited by 10 papers in the map (data through 13 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 48 of its 62 references that are in the map. First posted on arXiv 9 Sep 2024.

Builds on

48 of 62 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignment7 Apr
  2. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training22 Feb
  3. 2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution18 Dec
  4. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  5. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model3 Oct
  6. 2023Reward-Directed Conditional Diffusion: Provable Distribution Estimation and Reward Improvement13 Jul
  7. 2023Extracting Reward Functions from Diffusion Models1 Jun
  8. 2023MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL31 May
  9. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning29 May
  10. 2023Training Diffusion Models with Reinforcement Learning22 May
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  12. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
3 more
  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  2. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  3. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov

Other references · 33

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  4. 2020Denoising Diffusion Implicit Modelscited by 9,718
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  6. 2022Classifier-Free Diffusion Guidancecited by 5,785
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,656
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,751
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  10. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,974
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,415
  12. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,231
21 more
  1. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,176
  2. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,175
  3. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,035
  4. 2023Learning Universal Policies via Text-Guided Video Generationcited by 515
  5. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 488
  6. 2023Learning Interactive Real-World Simulatorscited by 445
  7. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 316
  8. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  9. 2023TextDiffuser: Diffusion Models as Text Painterscited by 224
  10. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  11. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  12. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 78
  13. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  14. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  15. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  16. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  17. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  18. 2022Models of human preference for learning reward functionscited by 54
  19. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  20. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  21. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 20

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 5.

10 papers.

  1. 2024Radiology Report Generation via Multi-objective Preference Optimizationcited by 15
  2. 2024Preference Aligned Diffusion Planner for Quadrupedal Locomotion Controlcited by 7
  3. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?cited by 2
  4. 2026RoDiF: Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedbackcited by 2
  5. 2026f-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analysescited by 2
  6. 2026Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrabilitycited by 1
  7. 2025Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineeringcited by 0
  8. 2025PC-Diffusion: Aligning Diffusion Models with Human Preferences via Preference Classifiercited by 0
  9. 2026From Noise to Control: Parameterized Diffusion Policiescited by 0
  10. 2026Latent evolving World Action Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Data-regularized Reinforcement Learning for Diffusion Models at Scale
  2. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models
  3. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control
  4. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  5. 2026Towards General Preference Alignment: Diffusion Models at Nash Equilibrium
  6. 2026iADD: Improving Alignment and Diversity in Diffusion Policy Optimization
  7. 2025Reinforcing Diffusion Models by Direct Group Preference Optimization
  8. 2026Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View