mlsanityData to 14 Oct 2026

Safe Offline Reinforcement Learning with Real-Time Budget Constraints

arXiv 2306.00603 ↗First posted 1 Jun 2023cs.LG cs.AI cs.ROcited by 28

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Aiming at promoting the safe real-world deployment of Reinforcement Learning (RL), research on safe RL has made significant progress in recent years. However, most existing works in the literature still focus on the online setting where risky violations of the safety budget are likely to be incurred during training. Besides, in many real-world applications, the learned policy is required to respond to dynamically determined safety budgets (i.e., constraint threshold) in real time. In this paper, we target at the above real-time budget constraint problem under the offline setting, and propose Trajectory-based REal-time Budget Inference (TREBI) as a novel solution that models this problem from the perspective of trajectory distribution and solves it through diffusion model planning. Theoretically, we prove an error bound of the estimation on the episodic reward and cost under the offline setting and thus provide a performance guarantee for TREBI. Empirical results on a wide range of simulation tasks and a real-world large-scale advertising application demonstrate the capability of TREBI in solving real-time budget constraint problems under offline settings.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 40 of its 54 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

40 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning14 Feb
  2. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning19 Jul

Other references · 38

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2016OpenAI Gymcited by 1,736
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  10. 2017Constrained Policy Optimizationcited by 983
  11. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
26 more
  1. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  2. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  3. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  4. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  7. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  8. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  9. 2018Reward Constrained Policy Optimizationcited by 417
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  11. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  12. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  13. 2019Batch Policy Learning under Constraintscited by 244
  14. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 206
  15. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  16. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  17. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  18. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  19. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
  20. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
  21. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  22. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 69
  23. 2022Constrained Policy Optimization via Bayesian World Modelscited by 57
  24. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  25. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 36
  26. 2022CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learningcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 9; 2026: 10.

28 papers.

Sort:

By area:

  1. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 64
  2. 2023Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learningcited by 18
  3. 2024Off-Policy Primal-Dual Safe Reinforcement Learningcited by 14
  4. 2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learningcited by 13
  5. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  6. 2025One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learningcited by 8
  7. 2024Cooperative Backdoor Attack in Decentralized Reinforcement Learning with Theoretical Guaranteecited by 6
  8. 2024Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policycited by 5
  9. 2024An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learningcited by 5
  10. 2025V-OCBF: Learning Safety Filters from Offline Data via Value-Guided Offline Control Barrier Functionscited by 5
  11. 2026Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policiescited by 4
  12. 2024FOSP: Fine-tuning Offline Safe Policy through World Modelscited by 3
16 more
  1. 2025From Uncertain to Safe: Conformal Adaptation of Diffusion Models for Safe PDE Controlcited by 3
  2. 2025A Provable Approach for End-to-End Safe Reinforcement Learningcited by 2
  3. 2025Expert-Guided Diffusion Planner for Auto-Biddingcited by 2
  4. 2026Epigraph-Guided Flow Matching for Safe and Performant Offline Reinforcement Learningcited by 2
  5. 2025Semi-gradient DICE for Offline Constrained Reinforcement Learningcited by 1
  6. 2025Online Optimization for Offline Safe Reinforcement Learningcited by 1
  7. 2025Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RLcited by 0
  8. 2025Learning Multi-Timescale Interventions under Safety and Resource Constraintscited by 0
  9. 2026Conditional Sequence Modeling for Safe Reinforcement Learningcited by 0
  10. 2026Decoupled Guidance Diffusion for Adaptive Offline Safe Reinforcement Learningcited by 0
  11. 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learningcited by 0
  12. 2026Temporal Self-Imitation Learningcited by 0
  13. 2026Generative Optimization for Incentivized Advertising with Global Level Constraintscited by 0
  14. 2026Q-learning Penalized Transformer for Safe Offline Reinforcement Learningcited by 0
  15. 2026A Safe Action Is Not Enough: Feasible-Future Decoding for Vision-Language-Action Policiescited by 0
  16. 2026Reachability-Aware Diffusion Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Offline Safe Reinforcement Learning Using Trajectory Classification
  2. 2023SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning
  3. 2026Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning
  4. 2022Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation
  5. 2024Safe Reinforcement Learning for Constrained Markov Decision Processes with Stochastic Stopping Time
  6. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
  7. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  8. 2023Bayesian Inverse Transition Learning for Offline Settings