mlsanityData to 14 Oct 2026

Safe Policy Improvement by Minimizing Robust Baseline Regret

arXiv 1607.03842 ↗First posted 13 Jul 2016stat.MLcited by 53

Not on the map. Why?

Cited by 26 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

An important problem in sequential decision-making under uncertainty is to use limited data to compute a safe policy, i.e., a policy that is guaranteed to perform at least as well as a given baseline strategy. In this paper, we develop and analyze a new model-based approach to compute a safe policy when we have access to an inaccurate dynamics model of the system with known accuracy guarantees. Our proposed robust method uses this (inaccurate) model to directly minimize the (negative) regret w.r.t. the baseline policy. Contrary to the existing approaches, minimizing the regret allows one to improve the baseline policy in states with accurate dynamics and seamlessly fall back to the baseline policy, otherwise. We show that our formulation is NP-hard and propose an approximate algorithm. Our empirical results on several domains show that even this relatively simple approximate algorithm can significantly outperform standard approaches.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Jul 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 5; 2021: 9; 2022: 8; 2023: 13; 2024: 8; 2025: 6; 2026: 3.

53 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  5. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  6. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 111
  7. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  8. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  9. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 38
  10. 2020Bayesian Robust Optimization for Imitation Learningcited by 28
  11. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  12. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPscited by 22
38 more
  1. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weightingcited by 22
  2. 2024Robust Markov Decision Processes: A Place Where AI and Formal Methods Meetcited by 18
  3. 2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
  4. 2022Behavior Prior Representation learning for Offline Reinforcement Learningcited by 12
  5. 2023Safe Policy Improvement for POMDPs via Finite-State Controllerscited by 12
  6. 2023Solving Long-run Average Reward Robust MDPs via Stochastic Gamescited by 10
  7. 2023Decision-Making Under Uncertainty: Beyond Probabilitiescited by 9
  8. 2023More for Less: Safe Policy Improvement With Stronger Performance Guaranteescited by 9
  9. 2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 7
  10. 2022Safe Policy Improvement Approaches on Discrete Markov Decision Processescited by 5
  11. 2023Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraintscited by 5
  12. 2023Bi-Level Offline Policy Optimization with Limited Explorationcited by 5
  13. 2024What Are the Odds? Improving the foundations of Statistical Model Checkingcited by 5
  14. 2022Policy Learning for Robust Markov Decision Process with a Mismatched Generative Modelcited by 4
  15. 2022Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learningcited by 4
  16. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learningcited by 4
  17. 2024A Single-Loop Robust Policy Gradient Method for Robust Markov Decision Processescited by 4
  18. 2021Learning to Be Cautiouscited by 3
  19. 2022A Joint Imitation-Reinforcement Learning Framework for Reduced Baseline Regretcited by 3
  20. 2025Deep SPI: Safe Policy Improvement via World Modelscited by 3
  21. 2021Interpretable performance analysis towards offline reinforcement learning: A dataset perspectivecited by 2
  22. 2024Sequential Decision-Making under Uncertainty: A Robust MDPs reviewcited by 2
  23. 2024CSPI-MT: Calibrated Safe Policy Improvement with Multiple Testing for Threshold Policiescited by 2
  24. 2024Decision-Point Guided Safe Policy Improvementcited by 2
  25. 2025Beyond Non-Expert Demonstrations: Outcome-Driven Action Constraint for Offline Reinforcement Learningcited by 2
  26. 2025Data-Efficient Safe Policy Improvement Using Parametric Structurecited by 2
  27. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learningcited by 2
  28. 2021Greedy UnMixing for Q-Learning in Multi-Agent Reinforcement Learningcited by 1
  29. 2022Safe Policy Improvement Approaches and their Limitationscited by 1
  30. 2023Stackelberg Batch Policy Learningcited by 1
  31. 2023Efficient Open-world Reinforcement Learning via Knowledge Distillation and Autonomous Rule Discoverycited by 1
  32. 2024Sparsity-based Safety Conservatism for Constrained Offline Reinforcement Learningcited by 1
  33. 2025Seldonian Reinforcement Learning for Ad Hoc Teamworkcited by 1
  34. 2021An Offline Risk-aware Policy Selection Method for Bayesian Markov Decision Processescited by 0
  35. 2022Don't do it: Safer Reinforcement Learning With Rule-based Guidancecited by 0
  36. 2023Solving Robust MDPs through No-Regret Dynamicscited by 0
  37. 2023Conservative Exploration for Policy Optimization via Off-Policy Policy Evaluationcited by 0
  38. 2024Percentile Criterion Optimization in Offline Reinforcement Learningcited by 0

50 most cited of 53 All 53 (JSON)