mlsanityData to 7 Oct 2026

Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning

arXiv 2502.04778 ↗First posted 7 Feb 2025cs.LG cs.AIcited by 18

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Behavior regularization, which constrains the policy to stay close to some behavior policy, is widely used in offline reinforcement learning (RL) to manage the risk of hazardous exploitation of unseen actions. Nevertheless, existing literature on behavior-regularized RL primarily focuses on explicit policy parameterizations, such as Gaussian policies. Consequently, it remains unclear how to extend this framework to more advanced policy parameterizations, such as diffusion models. In this paper, we introduce BDPO, a principled behavior-regularized RL framework tailored for diffusion-based policies, thereby combining the expressive power of diffusion policies and the robustness provided by regularization. The key ingredient of our method is to calculate the Kullback-Leibler (KL) regularization analytically as the accumulated discrepancies in reverse-time transition kernels along the diffusion trajectory. By integrating the regularization, we develop an efficient two-time-scale actor-critic RL algorithm that produces the optimal policy while respecting the behavior constraint. Comprehensive evaluations conducted on synthetic 2D tasks and continuous control tasks from the D4RL benchmark validate its effectiveness and superior performance.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 46 of its 52 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

46 of 52 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Diffusion Policy Policy Optimization1 Sep
  2. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning29 Jul
  3. 2024Diffusion Spectral Representation for Reinforcement Learning23 Jun
  4. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning31 May
  5. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning30 May
  6. 2024Diffusion Actor-Critic with Entropy Regulator24 May
  7. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning6 Feb
  8. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  9. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  10. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  12. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
1 more
  1. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 33

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2016OpenAI Gymcited by 1,684
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
21 more
  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  7. 2022Transformers are Sample-Efficient World Modelscited by 317
  8. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  9. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  10. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  11. 2019Soft Actor-Critic for Discrete Action Settingscited by 161
  12. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  13. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  14. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  15. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  16. 2024Iterated Denoising Energy Matching for Sampling from Boltzmann Densitiescited by 113
  17. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  18. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
  19. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learningcited by 33
  20. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
  21. 2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learningcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 13.

18 papers.

Sort:

By area:

  1. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  2. 2025Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timestepscited by 6
  3. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 4
  4. 2025Adaptive Diffusion Policy Optimization for Robotic Manipulationcited by 3
  5. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policiescited by 2
  6. 2026Drift Q-Learningcited by 2
  7. 2026Reversal Q-Learningcited by 2
  8. 2026Contractive Diffusion Policies: Robust Action Diffusion via Contractive Score-Based Sampling with Differential Equationscited by 1
  9. 2026Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Drivingcited by 1
  10. 2026Improving Diffusion Planners by Self-Supervised Action Gating with Energiescited by 1
  11. 2026NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Modelscited by 1
  12. 2026GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learningcited by 1
6 more
  1. 2026Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learningcited by 1
  2. 2025Decision Flow Policy Optimizationcited by 0
  3. 2025D2 Actor Critic: Diffusion Actor Meets Distributional Criticcited by 0
  4. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learningcited by 0
  5. 2026Language-Critique Imitation Learning from Suboptimal Demonstrationscited by 0
  6. 2026Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  2. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  3. 2021Offline Reinforcement Learning with Soft Behavior Regularization
  4. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learning
  5. 2021BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning
  6. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  7. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
  8. 2023A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning