mlsanityData to 7 Oct 2026

Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learning

arXiv 2112.07701 ↗First posted 14 Dec 2021cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Reinforcement Learning (RL) agents in the real world must satisfy safety constraints in addition to maximizing a reward objective. Model-based RL algorithms hold promise for reducing unsafe real-world actions: they may synthesize policies that obey all constraints using simulated samples from a learned model. However, imperfect models can result in real-world constraint violations even for actions that are predicted to satisfy all constraints. We propose Conservative and Adaptive Penalty (CAP), a model-based safe RL framework that accounts for potential modeling errors by capturing model uncertainty and adaptively exploiting it to balance the reward and the cost objectives. First, CAP inflates predicted costs using an uncertainty-based penalty. Theoretically, we show that policies that satisfy this conservative cost constraint are guaranteed to also be feasible in the true environment. We further show that this guarantees the safety of all intermediate solutions during RL training. Further, CAP adaptively tunes this penalty during training using true cost feedback from the environment. We evaluate this conservative and adaptive penalty-based approach for model-based safe RL extensively on state and image-based environments. Our results demonstrate substantial gains in sample-efficiency while incurring fewer violations than prior safe RL algorithms. Code is available at: https://github.com/Redrew/CAP

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 21 of its 39 references that are in the map. First posted on arXiv 14 Dec 2021.

Builds on

21 of 39 references.

References · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2016OpenAI Gymcited by 1,684
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2017Constrained Policy Optimizationcited by 928
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  7. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  8. 2018Reward Constrained Policy Optimizationcited by 397
  9. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  10. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  11. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  12. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
9 more
  1. 2018Safe Exploration in Continuous Action Spacescited by 237
  2. 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 186
  3. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
  4. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  5. 2017Uncertainty-Aware Reinforcement Learning for Collision Avoidancecited by 86
  6. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  7. 2019Robust Model Predictive Shielding for Safe Reinforcement Learning with Stochastic Dynamicscited by 45
  8. 2020Cautious Adaptation For Reinforcement Learning in Safety-Critical Settingscited by 42
  9. 2019Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learningcited by 16

Built on by

Cited by year

Citing papers per year: 2022: 7; 2023: 3; 2024: 3.

13 papers.

Sort:

By area:

  1. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  2. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
  3. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  4. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  5. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  6. 2022User-Interactive Offline Reinforcement Learningcited by 11
  7. 2024Verified Safe Reinforcement Learning for Neural Network Dynamic Modelscited by 7
  8. 2022Effects of Safety State Augmentation on Safe Explorationcited by 5
  9. 2022Safe Reinforcement Learning with Contrastive Risk Predictioncited by 3
  10. 2022Guiding Safe Exploration with Weakest Preconditionscited by 3
  11. 2022Guided Safe Shooting: model based reinforcement learning with safety constraintscited by 1
  12. 2024Anomalous State Sequence Modeling to Enhance Safety in Reinforcement Learningcited by 1
1 more
  1. 2024GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Modelcited by 0