mlsanityData to 14 Oct 2026

Decision-Point Guided Safe Policy Improvement

arXiv 2410.09361 ↗First posted 12 Oct 2024cs.LGcited by 2

On the map: Offline RL and imitation learning from data → Safe offline RL

also in 1 more
Abstract (arXiv)

Within batch reinforcement learning, safe policy improvement (SPI) seeks to ensure that the learnt policy performs at least as well as the behavior policy that generated the dataset. The core challenge in SPI is seeking improvements while balancing risk when many state-action pairs may be infrequently visited. In this work, we introduce Decision Points RL (DPRL), an algorithm that restricts the set of state-action pairs (or regions for continuous states) considered for improvement. DPRL ensures high-confidence improvement in densely visited states (i.e. decision points) while still utilizing data from sparsely visited states. By appropriately limiting where and how we may deviate from the behavior policy, we achieve tighter bounds than prior work; specifically, our data-dependent bounds do not scale with the size of the state and action spaces. In addition to the analysis, we demonstrate that DPRL is both safe and performant on synthetic and real datasets.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 40 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

23 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023More for Less: Safe Policy Improvement With Stronger Performance Guarantees13 May
  2. 2022Safe Policy Improvement Approaches and their Limitations1 Aug
  3. 2019Safe Policy Improvement with Soft Baseline Bootstrapping11 Jul

Other references · 20

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  3. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  5. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  7. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  8. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
  9. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
  10. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  11. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 89
  12. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 74
8 more
  1. 2019Uniform Error Bounds for Gaussian Process Regression with Application to Safe Controlcited by 64
  2. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 53
  3. 2019Combining Parametric and Nonparametric Models for Off-Policy Evaluationcited by 17
  4. 2020DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPscited by 16
  5. 2023Model-based Offline Reinforcement Learning with Count-based Conservatismcited by 16
  6. 2022Model-based Trajectory Stitching for Improved Offline Reinforcement Learningcited by 15
  7. 2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraintscited by 13
  8. 2021Learning-to-defer for sequential medical decision-making under uncertaintycited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Strategically Linked Decisions in Long-Term Planning and Reinforcement Learningcited by 0
  2. 2026Bellman-Certified Rounding for Sparse Policy Deployment in MDPscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Provably Good Batch Reinforcement Learning Without Great Exploration
  2. 2021Continuous Doubly Constrained Batch Reinforcement Learning
  3. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
  4. 2019Safe Policy Improvement with an Estimated Baseline Policy
  5. 2025Provably Optimal Reinforcement Learning under Safety Filtering
  6. 2023Safe Policy Improvement for POMDPs via Finite-State Controllers
  7. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning
  8. 2025Deep SPI: Safe Policy Improvement via World Models