mlsanityData to 7 Oct 2026

Provably Safe Reinforcement Learning with Step-wise Violation Constraints

arXiv 2302.06064 ↗First posted 13 Feb 2023cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate a novel safe reinforcement learning problem with step-wise violation constraints. Our problem differs from existing works in that we consider stricter step-wise violation constraints and do not assume the existence of safe actions, making our formulation more suitable for safety-critical applications which need to ensure safety in all decision steps and may not always possess safe actions, e.g., robot control and autonomous driving. We propose a novel algorithm SUCBVI, which guarantees widetildeO(√ST) step-wise violation and widetildeO(√H3SAT) regret. Lower bounds are provided to validate the optimality in both violation and regret performance with respect to S and T. Moreover, we further study a novel safe reward-free exploration problem with step-wise violation constraints. For this problem, we design an (varepsilon,δ)-PAC algorithm SRF-UCRL, which achieves nearly state-of-the-art sample complexity widetildeO((fracS2AH2varepsilon+fracH4SAvarepsilon2)(log(frac1δ)+S)), and guarantees widetildeO(√ST) violation during the exploration. The experimental results demonstrate the superiority of our algorithms in safety performance, and corroborate our theoretical results.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 13 Feb 2023.

Builds on

35 of 45 references.

References · 35

  1. 2017Constrained Policy Optimizationcited by 928
  2. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  3. 2017Safe Reinforcement Learning via Shieldingcited by 427
  4. 2018Reward Constrained Policy Optimizationcited by 397
  5. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  6. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  7. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  8. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  9. 2019Batch Policy Learning under Constraintscited by 242
  10. 2018Safe Exploration in Continuous Action Spacescited by 237
  11. 2021Reinforcement learning based recommender systems: A surveycited by 187
  12. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 180
23 more
  1. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  2. 2020Exploration-Exploitation in Constrained MDPscited by 146
  3. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 131
  4. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  5. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 109
  6. 2019Non-Asymptotic Gap-Dependent Regret Bounds for Tabular MDPscited by 104
  7. 2020Fast active learning for pure exploration in reinforcement learningcited by 92
  8. 2021Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPscited by 78
  9. 2020Adaptive Reward-Free Explorationcited by 77
  10. 2020Safe Reinforcement Learning via Curriculum Inductioncited by 73
  11. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  12. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 55
  13. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 54
  14. 2022Reachability Constrained Reinforcement Learningcited by 52
  15. 2020Upper Confidence Primal-Dual Reinforcement Learning for CMDP with Adversarial Losscited by 46
  16. 2016On Lower Bounds for Regret in Reinforcement Learningcited by 39
  17. 2020A Sample-Efficient Algorithm for Episodic Finite-Horizon MDP with Constraintscited by 34
  18. 2021Safe Reinforcement Learning with Linear Function Approximationcited by 33
  19. 2021A Simple Reward-free Approach to Constrained Reinforcement Learningcited by 28
  20. 2020Learning in Markov Decision Processes under Constraintscited by 20
  21. 2021Provably Efficient Algorithms for Multi-Objective Competitive RLcited by 20
  22. 2023A Near-Optimal Algorithm for Safe Reinforcement Learning Under Instantaneous Hard Constraintscited by 15
  23. 2022Safe Exploration Incurs Nearly No Additional Sample Complexity for Reward-free RLcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 2; 2026: 1.

10 papers.

  1. 2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 33
  2. 2024Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learningcited by 11
  3. 2023State-Wise Safe Reinforcement Learning With Pixel Observationscited by 9
  4. 2025One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learningcited by 8
  5. 2024Verified Safe Reinforcement Learning for Neural Network Dynamic Modelscited by 7
  6. 2024Leveraging Approximate Model-based Shielding for Probabilistic Safety Guarantees in Continuous Environmentscited by 5
  7. 2024Flipping-based Policy for Chance-Constrained Markov Decision Processescited by 3
  8. 2024SaVeR: Optimal Data Collection Strategy for Safe Policy Evaluation in Tabular MDPcited by 1
  9. 2025Guardian: Decoupling Exploration from Safety in Reinforcement Learningcited by 0
  10. 2026WINFlowNets: Warm-up Integrated Networks Training of Generative Flow Networks for Robotics and Machine Fault Adaptationcited by 0