mlsanityData to 7 Oct 2026

Effects of Safety State Augmentation on Safe Exploration

arXiv 2206.02675 ↗First posted 6 Jun 2022cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Safe exploration is a challenging and important problem in model-free reinforcement learning (RL). Often the safety cost is sparse and unknown, which unavoidably leads to constraint violations -- a phenomenon ideally to be avoided in safety-critical applications. We tackle this problem by augmenting the state-space with a safety state, which is nonnegative if and only if the constraint is satisfied. The value of this state also serves as a distance toward constraint violation, while its initial value indicates the available safety budget. This idea allows us to derive policies for scheduling the safety budget during training. We call our approach Simmer (Safe policy IMproveMEnt for RL) to reflect the careful nature of these schedules. We apply this idea to two safe RL problems: RL with constraints imposed on an average cost, and RL with constraints imposed on a cost with probability one. Our experiments suggest that "simmering, a safe algorithm can improve safety during training for both settings. We further show that Simmer can stabilize training and improve the performance of safe RL with average constraints.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 33 of its 67 references that are in the map. First posted on arXiv 6 Jun 2022.

Builds on

33 of 67 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2017Constrained Policy Optimizationcited by 928
  7. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  8. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  9. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  10. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  11. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
  12. 2018Safe Exploration in Continuous Action Spacescited by 237
21 more
  1. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  2. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 150
  3. 2018Learning-based Model Predictive Control for Safe Explorationcited by 116
  4. 2020Conservative Safety Critics for Explorationcited by 107
  5. 2017Uncertainty-Aware Reinforcement Learning for Collision Avoidancecited by 86
  6. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  7. 2020Safe Reinforcement Learning via Curriculum Inductioncited by 73
  8. 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 72
  9. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  10. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63
  11. 2022Constrained Policy Optimization via Bayesian World Modelscited by 55
  12. 2017Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Controlcited by 50
  13. 2020Cautious Adaptation For Reinforcement Learning in Safety-Critical Settingscited by 42
  14. 2016Safe Learning of Regions of Attraction for Uncertain, Nonlinear Systems with Gaussian Processescited by 38
  15. 2018Safely Learning to Control the Constrained Linear Quadratic Regulatorcited by 31
  16. 2020SAMBA: Safe Model-Based & Active Reinforcement Learningcited by 28
  17. 2018Barrier-Certified Adaptive Reinforcement Learning with Applications to Brushbot Navigationcited by 15
  18. 2021Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learningcited by 13
  19. 2021Reinforcement Learning with Almost Sure Constraintscited by 11
  20. 2021DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Interventioncited by 4
  21. 2020SafePILCO: a software tool for safe and data-efficient policy synthesiscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 4.

5 papers.

  1. 2026Extreme Value Policy Optimization for Safe Reinforcement Learningcited by 3
  2. 2026Off-Policy Safe Reinforcement Learning with Constrained Optimistic Explorationcited by 2
  3. 2024Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraintscited by 1
  4. 2026Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Explorationcited by 1
  5. 2026Utility-Constrained Policy Optimizationcited by 0