mlsanityData to 7 Oct 2026

Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk

arXiv 2206.04436 ↗First posted 9 Jun 2022cs.LG cs.AIcited by 40

Not on the map. Why?

Abstract (arXiv)

Though deep reinforcement learning (DRL) has obtained substantial success, it may encounter catastrophic failures due to the intrinsic uncertainty of both transition and observation. Most of the existing methods for safe reinforcement learning can only handle transition disturbance or observation disturbance since these two kinds of disturbance affect different parts of the agent; besides, the popular worst-case return may lead to overly pessimistic policies. To address these issues, we first theoretically prove that the performance degradation under transition disturbance and observation disturbance depends on a novel metric of Value Function Range (VFR), which corresponds to the gap in the value function between the best state and the worst state. Based on the analysis, we adopt conditional value-at-risk (CVaR) as an assessment of risk and propose a novel reinforcement learning algorithm of CVaR-Proximal-Policy-Optimization (CPPO) which formalizes the risk-sensitive constrained optimization problem by keeping its CVaR under a given threshold. Experimental results show that CPPO achieves a higher cumulative reward and is more robust against both observation and transition disturbances on a series of continuous control tasks in MuJoCo.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 16 of its 47 references that are in the map. First posted on arXiv 9 Jun 2022.

Builds on

16 of 47 references.

References · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  8. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  9. 2017Adversarial Attacks on Neural Network Policiescited by 314
  10. 2018Learning to Drive in a Daycited by 266
  11. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
  12. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
4 more
  1. 2014Optimizing the CVaR via Samplingcited by 98
  2. 2014Algorithms for CVaR Optimization in MDPscited by 98
  3. 2021Risk-Averse Bayes-Adaptive Reinforcement Learningcited by 28
  4. 2013Scaling Up Robust MDPs by Reinforcement Learningcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 8; 2024: 12; 2025: 8; 2026: 9.

40 papers.

Sort:

By area:

  1. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
  2. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  3. 2024Constrained Reinforcement Learning with Smoothed Log Barrier Functioncited by 12
  4. 2024Safe Reinforcement Learning on the Constraint Manifold: Theory and Applicationscited by 12
  5. 2023Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Riskcited by 11
  6. 2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 9
  7. 2023Adjustable Robust Reinforcement Learning for Online 3D Bin Packingcited by 9
  8. 2024PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learningcited by 9
  9. 2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimizationcited by 7
  10. 2022Consistent Attack: Universal Adversarial Perturbation on Embodied Vision Navigationcited by 6
  11. 2023Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learningcited by 6
  12. 2023Learning Diverse Risk Preferences in Population-based Self-playcited by 6
28 more
  1. 2023Task Aware Dreamer for Task Generalization in Reinforcement Learningcited by 5
  2. 2023Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theorycited by 5
  3. 2024Spectral-Risk Safe Reinforcement Learning with Convergence Guaranteescited by 5
  4. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
  5. 2023TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learningcited by 4
  6. 2024Bridging the gap between Learning-to-plan, Motion Primitives and Safe Reinforcement Learningcited by 4
  7. 2025Exploratory Diffusion Model for Unsupervised Reinforcement Learningcited by 4
  8. 2025Adversarial Diffusion for Robust Reinforcement Learningcited by 4
  9. 2023SoK: Adversarial Machine Learning Attacks and Defences in Multi-Agent Reinforcement Learningcited by 3
  10. 2026Beyond Rewards in Reinforcement Learning for Cyber Defencecited by 3
  11. 2022On the Reuse Bias in Off-Policy Reinforcement Learningcited by 2
  12. 2024Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Riskcited by 2
  13. 2024Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learningcited by 2
  14. 2025Safety-Aware Reinforcement Learning for Control via Risk-Sensitive Action-Value Iteration and Quantile Regressioncited by 2
  15. 2024Bootstrapping Expectiles in Reinforcement Learningcited by 1
  16. 2025Reinforced Embodied Active Defense: Exploiting Adaptive Interaction for Robust Visual Perception in Adversarial 3D Environmentscited by 1
  17. 2026Safe Continual Reinforcement Learning Methods for Nonstationary Environments. Towards a Survey of the State of the Artcited by 1
  18. 2026Diffusion Model-based Reinforcement Learning for Version Age of Information Scheduling: Average and Tail-Risk-Sensitive Controlcited by 1
  19. 2024Revisiting Safe Exploration in Safe Reinforcement learningcited by 0
  20. 2025Risk-averse policies for natural gas futures trading using distributional reinforcement learningcited by 0
  21. 2025Measures of Variability for Risk-averse Policy Gradientcited by 0
  22. 2025Bridging Econometrics and AI: VaR Estimation via Reinforcement Learning and GARCH Modelscited by 0
  23. 2025Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluationcited by 0
  24. 2026Fuz-RL: A Fuzzy-Guided Robust Framework for Safe Reinforcement Learning under Uncertaintycited by 0
  25. 2026Safe-Support Q-Learning: Learning without Unsafe Explorationcited by 0
  26. 2026Stochastic Minimum-Cost Reach-Avoid Reinforcement Learningcited by 0
  27. 2026PACT: Self-Evolving Physical Safety Alignment for Diffusion Policies in Embodied Manipulationcited by 0
  28. 2026RS-Diffuser: Risk-Sensitive Diffusion Planning with Distributional Value Guidancecited by 0