mlsanityData to 14 Oct 2026

DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning

arXiv 2310.05333 ↗First posted 9 Oct 2023cs.LGcited by 28

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Constrained policy search (CPS) is a fundamental problem in offline reinforcement learning, which is generally solved by advantage weighted regression (AWR). However, previous methods may still encounter out-of-distribution actions due to the limited expressivity of Gaussian-based policies. On the other hand, directly applying the state-of-the-art models with distribution expression capabilities (i.e., diffusion models) in the AWR framework is intractable since AWR requires exact policy probability densities, which is intractable in diffusion models. In this paper, we propose a novel approach, Diffusion-based Constrained Policy Search (dubbed DiffCPS), which tackles the diffusion-based constrained policy search with the primal-dual method. The theoretical analysis reveals that strong duality holds for diffusion-based CPS problems, and upon introducing parameter approximation, an approximated solution can be obtained after O(1/ε) number of dual iterations, where ε denotes the representation ability of the parametrized policy. Extensive experimental results based on the D4RL benchmark demonstrate the efficacy of our approach. We empirically show that DiffCPS achieves better or at least competitive performance compared to traditional AWR-based baselines as well as recent diffusion-based offline RL methods. The code is now available at https://github.com/felix-thu/DiffCPS.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 51 of its 60 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

51 of 60 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  2. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  3. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  4. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners3 Feb
  5. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  6. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  7. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2022Generative Adversarial Networkscited by 5,627
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  10. 2013Auto-Encoding Variational Bayescited by 4,053
  11. 2015Continuous control with deep reinforcement learningcited by 3,407
  12. 2021Convex optimizationcited by 2,515
32 more
  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2015Variational Inference with Normalizing Flowscited by 2,170
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  11. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  13. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  14. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  15. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  16. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  17. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 516
  18. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 379
  19. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
  20. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  21. 2020Critic Regularized Regressioncited by 275
  22. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  23. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  24. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  25. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  26. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 115
  27. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  28. 2018Deep Imitative Models for Flexible Inference, Planning, and Controlcited by 76
  29. 2019Learning When-to-Treat Policiescited by 49
  30. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  31. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  32. 2023Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matchingcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 9; 2026: 11.

28 papers.

Sort:

By area:

  1. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  2. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 110
  3. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  4. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 73
  5. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 58
  6. 2024DiffuserLite: Towards Real-time Diffusion Planningcited by 42
  7. 2024CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Makingcited by 39
  8. 2025Flow-Based Policy for Online Reinforcement Learningcited by 37
  9. 2026Q-learning with Adjoint Matchingcited by 37
  10. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  11. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  12. 2024AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimizationcited by 19
16 more
  1. 2026Flow Policy Gradients for Robot Controlcited by 16
  2. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learningcited by 9
  3. 2025Unleashing Flow Policies with Distributional Criticscited by 8
  4. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  5. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 6
  6. 2024Enhancing Exploration with Diffusion Policies in Hybrid Off-Policy RL: Application to Non-Prehensile Manipulationcited by 3
  7. 2026QPILOTS: Efficient Test-Time Q-Steering for Flow Policiescited by 3
  8. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Developmentcited by 2
  9. 2026Entropy-Regularized Adjoint Matching for Offline Reinforcement Learningcited by 2
  10. 2026Dual-Flow Reinforcement Learning with State-Aware Explorationcited by 1
  11. 2026Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learningcited by 1
  12. 2024DiffPoGAN: Diffusion Policies with Generative Adversarial Networks for Offline Reinforcement Learningcited by 0
  13. 2025Forecasting in Offline Reinforcement Learning for Non-stationary Environmentscited by 0
  14. 2026RoboTALES: Learning Reasoning-Guided Robot Policies via Task-Aligned Simulated Futurescited by 0
  15. 2026VINE: Taming Generative Control Policies for Reinforcement Learningcited by 0
  16. 2026Reinforcement Learning via Advantage Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion Policies with Value-Conditional Optimization for Offline Reinforcement Learning
  2. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning
  3. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  4. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
  5. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
  6. 2026Diffusion Policy through Conditional Proximal Policy Optimization
  7. 2024Diffusion Spectral Representation for Reinforcement Learning
  8. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learning