mlsanityData to 14 Oct 2026

Conservative Exploration for Policy Optimization via Off-Policy Policy Evaluation

arXiv 2312.15458 ↗First posted 24 Dec 2023stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

A precondition for the deployment of a Reinforcement Learning agent to a real-world system is to provide guarantees on the learning process. While a learning algorithm will eventually converge to a good policy, there are no guarantees on the performance of the exploratory policies. We study the problem of conservative exploration, where the learner must at least be able to guarantee its performance is at least as good as a baseline policy. We propose the first conservative provably efficient model-free algorithm for policy optimization in continuous finite-horizon problems. We leverage importance sampling techniques to counterfactually evaluate the conservative condition from the data self-generated by the algorithm. We derive a regret bound and show that (w.h.p.) the conservative constraint is never violated during learning. Finally, we leverage these insights to build a general schema for conservative exploration in DeepRL via off-policy policy evaluation techniques. We show empirically the effectiveness of our methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 51 of its 78 references that are in the map. First posted on arXiv 24 Dec 2023.

Builds on

51 of 78 references.

References · 51

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  8. 2017Constrained Policy Optimizationcited by 983
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  10. 2016Deep Exploration via Bootstrapped DQNcited by 527
  11. 2018Is Q-learning Provably Efficient?cited by 487
  12. 2018Reward Constrained Policy Optimizationcited by 417
38 more
  1. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  2. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  3. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  4. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  5. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  6. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  7. 2019Batch Policy Learning under Constraintscited by 244
  8. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  9. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 200
  10. 2020Exploration-Exploitation in Constrained MDPscited by 149
  11. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  12. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  13. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  14. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  15. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  16. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  17. 2020Conservative Safety Critics for Explorationcited by 113
  18. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  19. 2019Linear Stochastic Bandits Under Safety Constraintscited by 90
  20. 2020Safe Reinforcement Learning via Curriculum Inductioncited by 74
  21. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  22. 2020Stochastic Bandits with Linear Constraintscited by 68
  23. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  24. 2016Conservative Banditscited by 59
  25. 2016Conservative Contextual Linear Banditscited by 57
  26. 2015Bayesian Incentive-Compatible Bandit Explorationcited by 53
  27. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 53
  28. 2016Q(λ) with Off-Policy Correctionscited by 48
  29. 2021Universal Off-Policy Evaluationcited by 43
  30. 2019Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Samplingcited by 37
  31. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  32. 2020Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluationcited by 28
  33. 2020Conservative Exploration in Reinforcement Learningcited by 20
  34. 2020Improved Algorithms for Conservative Exploration in Banditscited by 16
  35. 2021Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Boundscited by 11
  36. 2020A One-Size-Fits-All Solution to Conservative Bandit Problemscited by 5
  37. 2018Conservative Exploration using Interleavingcited by 4
  38. 2019Contextual Combinatorial Conservative Banditscited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.