mlsanityData to 7 Oct 2026

OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning

arXiv 2407.14653 ↗First posted 19 Jul 2024cs.LGcited by 10

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Offline safe reinforcement learning (RL) aims to train a policy that satisfies constraints using a pre-collected dataset. Most current methods struggle with the mismatch between imperfect demonstrations and the desired safe and rewarding performance. In this paper, we introduce OASIS (cOnditionAl diStributIon Shaping), a new paradigm in offline safe RL designed to overcome these critical limitations. OASIS utilizes a conditional diffusion model to synthesize offline datasets, thus shaping the data distribution toward a beneficial target domain. Our approach makes compliance with safety constraints through effective data utilization and regularization techniques to benefit offline safe RL training. Comprehensive evaluations on public benchmarks and varying datasets showcase OASIS's superiority in benefiting offline safe RL agents to achieve high-reward behavior while satisfying the safety constraints, outperforming established baselines. Furthermore, OASIS exhibits high data efficiency and robustness, making it suitable for real-world applications, particularly in tasks where safety is imperative and high-quality demonstrations are scarce.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 74 of its 87 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

74 of 87 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model19 Jan
  2. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration18 Sep
  3. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning15 Jun
  4. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraints1 Jun
  5. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning14 Feb
  6. 2023SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning28 Jan
  7. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning19 Jul

Other references · 67

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  12. 2017Constrained Policy Optimizationcited by 928
38 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  4. 2014Semi-Supervised Learning with Deep Generative Modelscited by 494
  5. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptionscited by 440
  6. 2018Reward Constrained Policy Optimizationcited by 397
  7. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  8. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  9. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  10. 2019Batch Policy Learning under Constraintscited by 242
  11. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  12. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  13. 2022Convergence for score-based generative modeling with polynomial complexitycited by 176
  14. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  15. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
  16. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  17. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  18. 2023Synthetic Experience Replaycited by 100
  19. 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
  20. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  21. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  22. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  23. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  24. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  25. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  26. 2023SafeDiffuser: Safe Planning with Diffusion Probabilistic Modelscited by 69
  27. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  28. 2020Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentationcited by 57
  29. 2024Agile But Safe: Learning Collision-Free High-Speed Legged Locomotioncited by 53
  30. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  31. 2023Reward-Directed Conditional Diffusion: Provable Distribution Estimation and Reward Improvementcited by 49
  32. 2022MoCoDA: Model-based Counterfactual Data Augmentationcited by 44
  33. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
  34. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  35. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  36. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
  37. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weightingcited by 22
  38. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 3.

10 papers.

  1. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learningcited by 7
  2. 2025Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Drivingcited by 7
  3. 2025Tailored Primitive Initialization is the Secret Key to Reinforcement Learningcited by 4
  4. 2024FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learningcited by 3
  5. 2025Boundary-to-Region Supervision for Offline Safe Reinforcement Learningcited by 1
  6. 2025Online Optimization for Offline Safe Reinforcement Learningcited by 1
  7. 2025Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RLcited by 0
  8. 2026GAS: Enhancing Reward-Cost Balance of Generative Model-assisted Offline Safe RLcited by 0
  9. 2026Conditional Sequence Modeling for Safe Reinforcement Learningcited by 0
  10. 2026Q-learning Penalized Transformer for Safe Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RL
  2. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
  3. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
  4. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage
  5. 2024FOSP: Fine-tuning Offline Safe Policy through World Models
  6. 2024Offline Safe Reinforcement Learning Using Trajectory Classification
  7. 2026Epigraph-Guided Flow Matching for Safe and Performant Offline Reinforcement Learning
  8. 2025RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularization