mlsanityData to 14 Oct 2026

Safe Policy Improvement with Baseline Bootstrapping

arXiv 1712.06924 ↗First posted 19 Dec 2017cs.LG cs.AI stat.MLcited by 144

Not on the map. Why?

Cited by 84 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper considers Safe Policy Improvement (SPI) in Batch Reinforcement Learning (Batch RL): from a fixed dataset and without direct access to the true environment, train a policy that is guaranteed to perform at least as well as the baseline policy used to collect the data. Our approach, called SPI with Baseline Bootstrapping (SPIBB), is inspired by the knows-what-it-knows paradigm: it bootstraps the trained policy with the baseline when the uncertainty is high. Our first algorithm, Pib-SPIBB, comes with SPI theoretical guarantees. We also implement a variant, Pi≤ b-SPIBB, that is even more efficient in practice. We apply our algorithms to a motivational stochastic gridworld domain and further demonstrate on randomly generated MDPs the superiority of SPIBB with respect to existing algorithms, not only in safety but also in mean performance. Finally, we implement a model-free version of SPIBB and show its benefits on a navigation task with deep RL implementation called SPIBB-DQN, which is, to the best of our knowledge, the first RL algorithm relying on a neural network representation able to train efficiently and reliably from batch data, without any interaction with the environment.

Key facts. Cited by 144 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 2; 2020: 11; 2021: 33; 2022: 28; 2023: 25; 2024: 9; 2025: 17; 2026: 19.

144 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  4. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  5. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  6. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  8. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  9. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  10. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  11. 2021Offline RL Without Off-Policy Evaluationcited by 151
  12. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
38 more
  1. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
  2. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  3. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  4. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 111
  5. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  6. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 110
  7. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 89
  8. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 79
  9. 2020Provably Good Batch Reinforcement Learning Without Great Explorationcited by 77
  10. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 75
  11. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 71
  12. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  13. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  14. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  15. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 62
  16. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
  17. 2021Corruption-Robust Offline Reinforcement Learningcited by 51
  18. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  19. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learningcited by 43
  20. 2022Modular Lifelong Reinforcement Learning via Neural Compositioncited by 40
  21. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  22. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RLcited by 36
  23. 2022Data-Driven Offline Decision-Making via Invariant Representation Learningcited by 31
  24. 2023Adversarial Model for Offline Reinforcement Learningcited by 31
  25. 2022Sustainable Online Reinforcement Learning for Auto-biddingcited by 29
  26. 2020Bayesian Robust Optimization for Imitation Learningcited by 28
  27. 2021Offline Reinforcement Learning with Pseudometric Learningcited by 28
  28. 2021Medical Dead-ends and Learning to Identify High-risk States and Treatmentscited by 28
  29. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  30. 2022Lyapunov Density Models: Constraining Distribution Shift in Learning-Based Controlcited by 28
  31. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  32. 2022Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimalitycited by 24
  33. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
  34. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 23
  35. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPscited by 22
  36. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Beliefcited by 22
  37. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weightingcited by 22
  38. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21

50 most cited of 144 All 144 (JSON)