mlsanityData to 7 Oct 2026

More for Less: Safe Policy Improvement With Stronger Performance Guarantees

arXiv 2305.07958 ↗First posted 13 May 2023cs.LG cs.AIcited by 9

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

In an offline reinforcement learning setting, the safe policy improvement (SPI) problem aims to improve the performance of a behavior policy according to which sample data has been generated. State-of-the-art approaches to SPI require a high number of samples to provide practical probabilistic guarantees on the improved policy's performance. We present a novel approach to the SPI problem that provides the means to require less data for such guarantees. Specifically, to prove the correctness of these guarantees, we devise implicit transformations on the data set and the underlying environment model that serve as theoretical foundations to derive tighter improvement bounds for SPI. Our empirical evaluation, using the well-established SPI with baseline bootstrapping (SPIBB) algorithm, on standard benchmarks shows that our method indeed significantly reduces the sample complexity of the SPIBB algorithm.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 19 of its 56 references that are in the map. First posted on arXiv 13 May 2023.

Builds on

19 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs31 May
  2. 2019Safe Policy Improvement with an Estimated Baseline Policy11 Sep
  3. 2019Safe Policy Improvement with Soft Baseline Bootstrapping11 Jul

Other references · 16

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  5. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  6. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  8. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  9. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  10. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
  11. 2020Towards Safe Policy Improvement for Non-Stationary MDPscited by 26
  12. 2020Robust Finite-State Controllers for Uncertain POMDPscited by 23
4 more
  1. 2023Safe Policy Improvement for POMDPs via Finite-State Controllerscited by 10
  2. 2022Safe Policy Improvement Approaches on Discrete Markov Decision Processescited by 5
  3. 2022Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learningcited by 4
  4. 2022Strategy Complexity of Point Payoff, Mean Payoff and Total Payoff Objectives in Countable MDPscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 4.

9 papers.

  1. 2024Robust Markov Decision Processes: A Place Where AI and Formal Methods Meetcited by 18
  2. 2024What Are the Odds? Improving the foundations of Statistical Model Checkingcited by 5
  3. 2025Deep SPI: Safe Policy Improvement via World Modelscited by 3
  4. 2026Formally Verifying and Explaining Sepsis Treatment Policies with COOL-MCcited by 3
  5. 2024Decision-Point Guided Safe Policy Improvementcited by 2
  6. 2025Data-Efficient Safe Policy Improvement Using Parametric Structurecited by 2
  7. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learningcited by 2
  8. 2026COOL-MC: Verifying and Explaining RL Policies for Platelet Inventory Managementcited by 0
  9. 2026Bellman-Certified Rounding for Sparse Policy Deployment in MDPscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Safe Policy Improvement Approaches and their Limitations
  2. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
  3. 2026Multi-step Proximal Policy Improvement in Offline Reinforcement Learning
  4. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
  5. 2026Policy Improvement Reinforcement Learning
  6. 2025Online Optimization for Offline Safe Reinforcement Learning
  7. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning
  8. 2021Offline RL Without Off-Policy Evaluation