mlsanityData to 7 Oct 2026

TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learning

arXiv 2312.00344 ↗First posted 1 Dec 2023cs.RO cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

As safety is of paramount importance in robotics, reinforcement learning that reflects safety, called safe RL, has been studied extensively. In safe RL, we aim to find a policy which maximizes the desired return while satisfying the defined safety constraints. There are various types of constraints, among which constraints on conditional value at risk (CVaR) effectively lower the probability of failures caused by high costs since CVaR is a conditional expectation obtained above a certain percentile. In this paper, we propose a trust region-based safe RL method with CVaR constraints, called TRC. We first derive the upper bound on CVaR and then approximate the upper bound in a differentiable form in a trust region. Using this approximation, a subproblem to get policy gradients is formulated, and policies are trained by iteratively solving the subproblem. TRC is evaluated through safe navigation tasks in simulations with various robots and a sim-to-real environment with a Jackal robot from Clearpath. Compared to other safe RL methods, the performance is improved by 1.93 times while the constraints are satisfied in all experiments.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 17 of its 26 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

17 of 26 references.

References · 17

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  3. 2017Constrained Policy Optimizationcited by 928
  4. 2018Reward Constrained Policy Optimizationcited by 397
  5. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  6. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  7. 2020Projection-Based Constrained Policy Optimizationcited by 195
  8. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  9. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 131
  10. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  11. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 109
  12. 2020Conservative Safety Critics for Explorationcited by 107
5 more
  1. 2014Algorithms for CVaR Optimization in MDPscited by 98
  2. 2019Risk-Averse Trust Region Optimization for Reward-Volatility Reductioncited by 53
  3. 2019Worst Cases Policy Gradientscited by 52
  4. 2022Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Riskcited by 40
  5. 2020Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotioncited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Riskcited by 2
  2. 2024Safe CoR: A Dual-Expert Approach to Integrating Imitation Learning and Safe Reinforcement Learning Using Constraint Rewardscited by 1
  3. 2025Measures of Variability for Risk-averse Policy Gradientcited by 0
  4. 2026Safe-Support Q-Learning: Learning without Unsafe Explorationcited by 0