# More for Less: Safe Policy Improvement With Stronger Performance Guarantees

arXiv: https://arxiv.org/abs/2305.07958 · HTML: https://mlsanity.com/paper/2305.07958/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 19 of its 56 references that are in the map.
- First posted on arXiv 13 May 2023.

## Placed in

- Offline RL and imitation learning from data / [Safe offline RL](https://mlsanity.com/offlinerl/safe-offline-rl/)

## Builds on (most cited first, 19 of 19)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [COMBO: Conservative Offline Model-Based Policy Optimization](https://mlsanity.com/paper/2102.08363/) (2021)
- [Safe Policy Improvement with Baseline Bootstrapping](https://mlsanity.com/paper/1712.06924/) (2017)
- [RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2204.12581/) (2022)
- [Safe Policy Improvement by Minimizing Robust Baseline Regret](https://mlsanity.com/paper/1607.03842/) (2016)
- [Towards Safe Policy Improvement for Non-Stationary MDPs](https://mlsanity.com/paper/2010.12645/) (2020)
- [Robust Finite-State Controllers for Uncertain POMDPs](https://mlsanity.com/paper/2009.11459/) (2020)
- [Safe Policy Improvement with Soft Baseline Bootstrapping](https://mlsanity.com/paper/1907.05079/) (2019)
- [Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs](https://mlsanity.com/paper/2106.00099/) (2021)
- [Safe Policy Improvement with an Estimated Baseline Policy](https://mlsanity.com/paper/1909.05236/) (2019)
- [Safe Policy Improvement for POMDPs via Finite-State Controllers](https://mlsanity.com/paper/2301.04939/) (2023)
- [Safe Policy Improvement Approaches on Discrete Markov Decision Processes](https://mlsanity.com/paper/2201.12175/) (2022)
- [Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learning](https://mlsanity.com/paper/2206.01085/) (2022)
- [Strategy Complexity of Point Payoff, Mean Payoff and Total Payoff Objectives in Countable MDPs](https://mlsanity.com/paper/2203.07079/) (2022)

## Built on by (most cited first, 9 of 9)

- [Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet](https://mlsanity.com/paper/2411.11451/) (2024)
- [What Are the Odds? Improving the foundations of Statistical Model Checking](https://mlsanity.com/paper/2404.05424/) (2024)
- [Deep SPI: Safe Policy Improvement via World Models](https://mlsanity.com/paper/2510.12312/) (2025)
- [Formally Verifying and Explaining Sepsis Treatment Policies with COOL-MC](https://mlsanity.com/paper/2602.14505/) (2026)
- [Decision-Point Guided Safe Policy Improvement](https://mlsanity.com/paper/2410.09361/) (2024)
- [Data-Efficient Safe Policy Improvement Using Parametric Structure](https://mlsanity.com/paper/2507.15532/) (2025)
- [Robust Probabilistic Shielding for Safe Offline Reinforcement Learning](https://mlsanity.com/paper/2605.10293/) (2026)
- [COOL-MC: Verifying and Explaining RL Policies for Platelet Inventory Management](https://mlsanity.com/paper/2603.02396/) (2026)
- [Bellman-Certified Rounding for Sparse Policy Deployment in MDPs](https://mlsanity.com/paper/2610.00325/) (2026)

## Abstract (arXiv)

In an offline reinforcement learning setting, the safe policy improvement (SPI) problem aims to improve the performance of a behavior policy according to which sample data has been generated. State-of-the-art approaches to SPI require a high number of samples to provide practical probabilistic guarantees on the improved policy's performance. We present a novel approach to the SPI problem that provides the means to require less data for such guarantees. Specifically, to prove the correctness of these guarantees, we devise implicit transformations on the data set and the underlying environment model that serve as theoretical foundations to derive tighter improvement bounds for SPI. Our empirical evaluation, using the well-established SPI with baseline bootstrapping (SPIBB) algorithm, on standard benchmarks shows that our method indeed significantly reduces the sample complexity of the SPIBB algorithm.