mlsanityData to 7 Oct 2026

Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraints

arXiv 2306.06265 ↗First posted 9 Jun 2023cs.LG cs.IT math.ITcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper investigates conservative exploration in reinforcement learning where the performance of the learning agent is guaranteed to be above a certain threshold throughout the learning process. It focuses on the tabular episodic Markov Decision Process (MDP) setting that has finite states and actions. With the knowledge of an existing safe baseline policy, an algorithm termed as StepMix is proposed to balance the exploitation and exploration while ensuring that the conservative constraint is never violated in each episode with high probability. StepMix features a unique design of a mixture policy that adaptively and smoothly interpolates between the baseline policy and the optimistic policy. Theoretical analysis shows that StepMix achieves near-optimal regret order as in the constraint-free setting, indicating that obeying the stringent episode-wise conservative constraint does not compromise the learning performance. Besides, a randomization-based EpsMix algorithm is also proposed and shown to achieve the same performance as StepMix. The algorithm design and theoretical analysis are further extended to the setting where the baseline policy is not given a priori but must be learned from an offline dataset, and it is proved that similar conservative guarantee and regret can be achieved if the offline dataset is sufficiently large. Experiment results corroborate the theoretical analysis and demonstrate the effectiveness of the proposed conservative exploration strategies.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 36 of its 65 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

36 of 65 references.

References · 36

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  3. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  4. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
  5. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  6. 2020Exploration-Exploitation in Constrained MDPscited by 146
  7. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  8. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 131
  9. 2020Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcited by 104
  10. 2020Fast active learning for pure exploration in reinforcement learningcited by 92
  11. 2019Linear Stochastic Bandits Under Safety Constraintscited by 90
  12. 2020Optimistic Policy Optimization with Bandit Feedbackcited by 79
24 more
  1. 2021Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPscited by 78
  2. 2020Safe Reinforcement Learning via Curriculum Inductioncited by 73
  3. 2020Stochastic Bandits with Linear Constraintscited by 68
  4. 2016Conservative Banditscited by 58
  5. 2016Conservative Contextual Linear Banditscited by 56
  6. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
  7. 2021Policy Optimization in Adversarial MDPs: Improved Exploration via Dilated Bonusescited by 51
  8. 2020Constrained Upper Confidence Reinforcement Learningcited by 46
  9. 2020Planning in Markov Decision Processes with Gap-Dependent Sample Complexitycited by 34
  10. 2020A Sample-Efficient Algorithm for Episodic Finite-Horizon MDP with Constraintscited by 34
  11. 2021DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learningcited by 34
  12. 2021Safe Reinforcement Learning with Linear Function Approximationcited by 33
  13. 2022Provably Efficient Model-Free Constrained RL with Linear Function Approximationcited by 29
  14. 2019Safe Linear Stochastic Banditscited by 27
  15. 2020Kernel-Based Reinforcement Learning: A Finite-Time Analysiscited by 23
  16. 2020Conservative Exploration in Reinforcement Learningcited by 20
  17. 2020Improved Algorithms for Conservative Exploration in Banditscited by 16
  18. 2021Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPscited by 16
  19. 2020Accelerating Safe Reinforcement Learning with Constraint-mismatched Policiescited by 11
  20. 2021A Reduction-Based Framework for Conservative Bandits and Reinforcement Learningcited by 9
  21. 2022Safe Exploration Incurs Nearly No Additional Sample Complexity for Reward-free RLcited by 9
  22. 2022ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Datacited by 9
  23. 2020A One-Size-Fits-All Solution to Conservative Bandit Problemscited by 5
  24. 2021Maximum Entropy Reinforcement Learning with Mixture Policiescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 0; 2025: 1; 2026: 1.

4 papers.

  1. 2023Offline Reinforcement Learning for Wireless Network Optimization with Mixture Datasetscited by 12
  2. 2026BAGEN: Are LLM Agents Budget-Aware?cited by 12
  3. 2023Advancing RAN Slicing with Offline Reinforcement Learningcited by 5
  4. 2025Off-policy Reinforcement Learning with Model-based Exploration Augmentationcited by 3