mlsanityData to 14 Oct 2026

Online Optimization for Offline Safe Reinforcement Learning

arXiv 2510.22027 ↗First posted 24 Oct 2025cs.LG cs.AI stat.MLcited by 1

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

We study the problem of Offline Safe Reinforcement Learning (OSRL), where the goal is to learn a reward-maximizing policy from fixed data under a cumulative cost constraint. We propose a novel OSRL approach that frames the problem as a minimax objective and solves it by combining offline RL with online optimization algorithms. We prove the approximate optimality of this approach when integrated with an approximate offline RL oracle and no-regret online optimization. We also present a practical approximation that can be combined with any offline RL algorithm, eliminating the need for offline policy evaluation. Empirical results on the DSRL benchmark demonstrate that our method reliably enforces safety constraints under stringent cost budgets, while achieving high rewards. The code is available at https://github.com/yassineCh/O3SRL.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 50 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

39 of 50 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning25 Dec
  2. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning19 Jul
  3. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration18 Sep
  4. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning15 Jun
  5. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraints1 Jun
  6. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning14 Feb
  7. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning19 Jul

Other references · 32

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2019Introduction to Online Convex Optimizationcited by 803
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  7. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  8. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  9. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  10. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  11. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  12. 2019Batch Policy Learning under Constraintscited by 244
20 more
  1. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  2. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  3. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  4. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  5. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  6. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  7. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 84
  8. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 74
  9. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  10. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  11. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 48
  12. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 44
  13. 2019Off-Policy Policy Gradient with State Distribution Correctioncited by 43
  14. 2022Bootstrapped Transformer for Offline Reinforcement Learningcited by 37
  15. 2024Offline Model-Based Optimization via Policy-Guided Gradient Searchcited by 19
  16. 2023Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Drivingcited by 17
  17. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyondcited by 6
  18. 2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learningcited by 4
  19. 2023VIPeR: Provably Efficient Algorithm for Offline RL with Neural Function Approximationcited by 3
  20. 2025On The Statistical Complexity of Offline Decision-Makingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SteinGate: Tail-Sensitive Safe Reinforcement Learning via Stein Discrepancycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
  2. 2025Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis
  3. 2025Safe Reinforcement Learning with Minimal Supervision
  4. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  5. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
  6. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
  7. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  8. 2024Offline Safe Reinforcement Learning Using Trajectory Classification