mlsanityData to 14 Oct 2026

Data-Efficient Safe Policy Improvement Using Parametric Structure

arXiv 2507.15532 ↗First posted 21 Jul 2025cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Safe offline RL

also in 1 more
Abstract (arXiv)

Safe policy improvement (SPI) is an offline reinforcement learning problem in which a new policy that reliably outperforms the behavior policy with high confidence needs to be computed using only a dataset and the behavior policy. Markov decision processes (MDPs) are the standard formalism for modeling environments in SPI. In many applications, additional information in the form of parametric dependencies between distributions in the transition dynamics is available. We make SPI more data-efficient by leveraging these dependencies through three contributions: (1) a parametric SPI algorithm that exploits known correlations between distributions to more accurately estimate the transition dynamics using the same amount of data; (2) a preprocessing technique that prunes redundant actions from the environment through a game-based abstraction; and (3) a more advanced preprocessing technique, based on satisfiability modulo theory (SMT) solving, that can identify more actions to prune. Empirical results and an ablation study show that our techniques increase the data efficiency of SPI by multiple orders of magnitude while maintaining the same reliability guarantees.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 18 of its 50 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

18 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023More for Less: Safe Policy Improvement With Stronger Performance Guarantees13 May
  2. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs31 May

Other references · 16

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2017Safe Reinforcement Learning via Shieldingcited by 449
  3. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  4. 2022Partially Observable Markov Decision Processes in Robotics: A Surveycited by 86
  5. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 53
  6. 2020Towards Safe Policy Improvement for Non-Stationary MDPscited by 26
  7. 2024Robust Markov Decision Processes: A Place Where AI and Formal Methods Meetcited by 18
  8. 2023Safe Policy Improvement for POMDPs via Finite-State Controllerscited by 12
  9. 2017Parametric Markov Chains: PCTL Complexity and Fraction-free Gaussian Eliminationcited by 11
  10. 2022Parameter Synthesis in Markov Models: A Gentle Surveycited by 10
  11. 2016Optimizing Expectation with Guarantees in POMDPs (Technical Report)cited by 7
  12. 2024Certifiably Robust Policies for Uncertain Parametric Environmentscited by 7
4 more
  1. 2022Safe Policy Improvement Approaches on Discrete Markov Decision Processescited by 5
  2. 2024What Are the Odds? Improving the foundations of Statistical Model Checkingcited by 5
  3. 2018Expectation Optimization with Probabilistic Guarantees in POMDPs with Discounted-sum Objectivescited by 4
  4. 2024The Existential Theory of the Reals as a Complexity Class: A Compendiumcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Robust Parameter Learning for Uncertain MDPscited by 1
  2. 2026When Does Action Credit Need Updating?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Safe Policy Improvement Approaches and their Limitations
  2. 2025Deep SPI: Safe Policy Improvement via World Models
  3. 2024Decision-Point Guided Safe Policy Improvement
  4. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
  5. 2019Safe Policy Improvement with Soft Baseline Bootstrapping
  6. 2026Multi-step Proximal Policy Improvement in Offline Reinforcement Learning
  7. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  8. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning