mlsanityData to 7 Oct 2026

Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RL

arXiv 2502.12391 ↗First posted 18 Feb 2025cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Constrained reinforcement learning (RL) seeks high-performance policies under safety constraints. We focus on an offline setting where the agent has only a fixed dataset -- common in realistic tasks to prevent unsafe exploration. To address this, we propose Diffusion-Regularized Constrained Offline Reinforcement Learning (DRCORL), which first uses a diffusion model to capture the behavioral policy from offline data and then extracts a simplified policy to enable efficient inference. We further apply gradient manipulation for safety adaptation, balancing the reward objective and constraint satisfaction. This approach leverages high-quality offline data while incorporating safety requirements. Empirical results show that DRCORL achieves reliable safety performance, fast inference, and strong reward outcomes across robot learning tasks. Compared to existing safe offline RL methods, it consistently meets cost limits and performs well with the same hyperparameters, indicating practical applicability in real-world scenarios.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 55 of its 70 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

55 of 70 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning25 Dec
  2. 2024Offline Safe Reinforcement Learning Using Trajectory Classification19 Dec
  3. 2024FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learning12 Dec
  4. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning19 Jul
  5. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model19 Jan
  6. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning15 Jun
  7. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraints1 Jun
  8. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning14 Feb
  9. 2023SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning28 Jan
  10. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning19 Jul

Other references · 45

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  10. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  12. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
33 more
  1. 2017Constrained Policy Optimizationcited by 928
  2. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  4. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  5. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  6. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  7. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  8. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  9. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  10. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  11. 2022Convergence of denoising diffusion models under the manifold hypothesiscited by 237
  12. 2020Projection-Based Constrained Policy Optimizationcited by 195
  13. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  14. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 131
  15. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
  16. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  17. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  18. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 75
  19. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  20. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  21. 2023SafeDiffuser: Safe Planning with Diffusion Probabilistic Modelscited by 69
  22. 2023Score Regularized Policy Optimization through Diffusion Behaviorcited by 67
  23. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learningcited by 41
  24. 2019Bootstrapping Upper Confidence Boundcited by 36
  25. 2021A Dual Approach to Constrained Markov Decision Processes with Entropy Regularizationcited by 33
  26. 2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 33
  27. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  28. 2021Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancementcited by 28
  29. 2024Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulationcited by 21
  30. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
  31. 2020Balancing Constraints and Rewards with Meta-Gradient D4PGcited by 18
  32. 2024Off-Policy Primal-Dual Safe Reinforcement Learningcited by 13
  33. 2024Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulationcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
  2. 2024Diffusion Models for Offline Multi-agent Reinforcement Learning with Safety Constraints
  3. 2024FOSP: Fine-tuning Offline Safe Policy through World Models
  4. 2025Online Optimization for Offline Safe Reinforcement Learning
  5. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration
  6. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
  7. 2026Individual Control Barrier Functions-Guided Diffusion Model for Safe Offline Multi-Agent Reinforcement Learning
  8. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuning