mlsanityData to 14 Oct 2026

DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learning

arXiv 2112.00885 ↗First posted 1 Dec 2021cs.LG cs.AIcited by 35

Not on the map. Why?

Cited by 20 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Safe reinforcement learning is extremely challenging--not only must the agent explore an unknown environment, it must do so while ensuring no safety constraint violations. We formulate this safe reinforcement learning (RL) problem using the framework of a finite-horizon Constrained Markov Decision Process (CMDP) with an unknown transition probability function, where we model the safety requirements as constraints on the expected cumulative costs that must be satisfied during all episodes of learning. We propose a model-based safe RL algorithm that we call Doubly Optimistic and Pessimistic Exploration (DOPE), and show that it achieves an objective regret tildeO(|S|√|A| K) without violating the safety constraints during learning, where |S| is the number of states, |A| is the number of actions, and K is the number of learning episodes. Our key idea is to combine a reward bonus for exploration (optimism) with a conservative constraint (pessimism), in addition to the standard optimistic model-based exploration. DOPE is not only able to improve the objective regret bound, but also shows a significant empirical performance improvement as compared to earlier optimism-pessimism approaches.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 31 of its 62 references that are in the map. First posted on arXiv 1 Dec 2021.

Builds on

31 of 62 references.

References · 31

  1. 2017Constrained Policy Optimizationcited by 983
  2. 2019Introduction to Online Convex Optimizationcited by 803
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  4. 2018Reward Constrained Policy Optimizationcited by 417
  5. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  6. 2020Projection-Based Constrained Policy Optimizationcited by 203
  7. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 176
  8. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  9. 2020Exploration-Exploitation in Constrained MDPscited by 149
  10. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  11. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 106
  12. 2017Online Convex Optimization with Stochastic Constraintscited by 95
19 more
  1. 2019Linear Stochastic Bandits Under Safety Constraintscited by 90
  2. 2019Learning Adversarial MDPs with Bandit Feedback and Unknown Transitioncited by 85
  3. 2021Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPscited by 80
  4. 2020Stochastic Bandits with Linear Constraintscited by 68
  5. 2017Online Convex Optimization with Time-Varying Constraintscited by 56
  6. 2018Online convex optimization for cumulative constraintscited by 47
  7. 2020Constrained Upper Confidence Reinforcement Learningcited by 46
  8. 2020A Sample-Efficient Algorithm for Episodic Finite-Horizon MDP with Constraintscited by 34
  9. 2020Learning with Safety Constraints: Sample Complexity of Reinforcement Learning for Constrained MDPscited by 32
  10. 2021An Efficient Pessimistic-Optimistic Algorithm for Stochastic Linear Bandits with General Constraintscited by 29
  11. 2020Stage-wise Conservative Linear Banditscited by 28
  12. 2019Safe Linear Stochastic Banditscited by 27
  13. 2020Learning in Markov Decision Processes under Constraintscited by 20
  14. 2019Safe Convex Learning under Uncertain Constraintscited by 17
  15. 2021Safe Online Convex Optimization with Unknown Linear Safety Constraintscited by 13
  16. 2016Throughput Optimal Decentralized Scheduling of Multi-Hop Networks with End-to-End Deadline Constraints: Unreliable Linkscited by 6
  17. 2015Trading Safety Versus Performance: Rapid Deployment of Robotic Swarms with Robust Performance Constraintscited by 2
  18. 2016Decentralized Control via Dynamic Stochastic Prices: The Independent System Operator Problemcited by 2
  19. 2020Stochastic Linear Bandits with Protected Subspacecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 9; 2025: 2; 2026: 15.

35 papers.

Sort:

By area:

  1. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 44
  2. 2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 28
  3. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 20
  4. 2024Distributionally Robust Constrained Reinforcement Learning under Strong Dualitycited by 15
  5. 2023A safe exploration approach to constrained Markov decision processescited by 10
  6. 2024Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Formcited by 10
  7. 2023Directional Optimism for Safe Linear Banditscited by 8
  8. 2024Constrained Reinforcement Learning Under Model Mismatchcited by 8
  9. 2023Cancellation-Free Regret Bounds for Lagrangian Approaches in Constrained Markov Decision Processescited by 7
  10. 2023Provably Learning Nash Policies in Constrained Markov Potential Gamescited by 7
  11. 2023Safe Posterior Sampling for Constrained MDPs with Bounded Constraint Violationcited by 6
  12. 2024A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guaranteescited by 6
23 more
  1. 2025Provably Efficient RL under Episode-Wise Safety in Constrained MDPs with Linear Function Approximationcited by 6
  2. 2026Safe Exploration via Policy Priorscited by 6
  3. 2023Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraintscited by 5
  4. 2024Safe Reinforcement Learning for Constrained Markov Decision Processes with Stochastic Stopping Timecited by 5
  5. 2024A Retention-Centric Framework for Continual Learning with Guaranteed Model Developmental Safetycited by 4
  6. 2024Pure Exploration for Constrained Best Mixed Arm Identification with a Fixed Budgetcited by 2
  7. 2026Near-Optimal Primal-Dual Algorithm for Learning Linear Mixture CMDPs with Adversarial Rewardscited by 2
  8. 2026Learning Weakly Communicating Average-Reward CMDPs: Strong Duality and Improved Regretcited by 2
  9. 2026Robust Shielding for Safe Reinforcement Learningcited by 2
  10. 2023Online Resource Allocation in Episodic Markov Decision Processescited by 1
  11. 2024Structured Reinforcement Learning for Media Streaming at the Wireless Edgecited by 1
  12. 2026Constrained Meta Reinforcement Learning with Provable Test-Time Safetycited by 1
  13. 2026Near-Optimal Sample Complexity for Online Constrained MDPscited by 1
  14. 2026Golden Handcuffs make safer AI agentscited by 1
  15. 2025A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Modelscited by 0
  16. 2026Provably Safe Reinforcement Learning for Stochastic Reach-Avoid Problems with Entropy Regularizationcited by 0
  17. 2026Bi-Level Online Provisioning and Scheduling with Switching Costs and Cross-Level Constraintscited by 0
  18. 2026MC-CPO: Mastery-Conditioned Constrained Policy Optimization for Pedagogically Safe Intelligent Tutoring Systemscited by 0
  19. 2026Primal-Dual Policy Optimization for Linear CMDPs with Adversarial Lossescited by 0
  20. 2026Sampling-Based Safe Reinforcement Learningcited by 0
  21. 2026Provably Safe Sim-to-Real Transfercited by 0
  22. 2026Nociception as a Control Primitive: Afferent Channels and Nociceptive Memory for Agents Deployed in One Bodycited by 0
  23. 2026Instance-Dependent Regret for CMDPs with Step-Wise Constraintscited by 0