mlsanityData to 7 Oct 2026

Safe Policy Improvement Approaches and their Limitations

arXiv 2208.00724 ↗First posted 1 Aug 2022cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Safe Policy Improvement (SPI) is an important technique for offline reinforcement learning in safety critical applications as it improves the behavior policy with a high probability. We classify various SPI approaches from the literature into two groups, based on how they utilize the uncertainty of state-action pairs. Focusing on the Soft-SPIBB (Safe Policy Improvement with Soft Baseline Bootstrapping) algorithms, we show that their claim of being provably safe does not hold. Based on this finding, we develop adaptations, the Adv-Soft-SPIBB algorithms, and show that they are provably safe. A heuristic adaptation, Lower-Approx-Soft-SPIBB, yields the best performance among all SPIBB algorithms in extensive experiments on two benchmarks. We also check the safety guarantees of the provably safe algorithms and show that huge amounts of data are necessary such that the safety bounds become useful in practice.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 9 of its 23 references that are in the map. First posted on arXiv 1 Aug 2022.

Builds on

9 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2019Safe Policy Improvement with an Estimated Baseline Policy11 Sep
  2. 2019Safe Policy Improvement with Soft Baseline Bootstrapping11 Jul

Other references · 7

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  4. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  5. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  6. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
  7. 2022Safe Policy Improvement Approaches on Discrete Markov Decision Processescited by 5

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Decision-Point Guided Safe Policy Improvementcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023More for Less: Safe Policy Improvement With Stronger Performance Guarantees
  2. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
  3. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
  4. 2023Safe Policy Improvement for POMDPs via Finite-State Controllers
  5. 2025Data-Efficient Safe Policy Improvement Using Parametric Structure
  6. 2025Deep SPI: Safe Policy Improvement via World Models
  7. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning
  8. 2025Online Optimization for Offline Safe Reinforcement Learning