mlsanityData to 7 Oct 2026

Spectral-Risk Safe Reinforcement Learning with Convergence Guarantees

arXiv 2405.18698 ↗First posted 29 May 2024cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

The field of risk-constrained reinforcement learning (RCRL) has been developed to effectively reduce the likelihood of worst-case scenarios by explicitly handling risk-measure-based constraints. However, the nonlinearity of risk measures makes it challenging to achieve convergence and optimality. To overcome the difficulties posed by the nonlinearity, we propose a spectral risk measure-constrained RL algorithm, spectral-risk-constrained policy optimization (SRCPO), a bilevel optimization approach that utilizes the duality of spectral risk measures. In the bilevel optimization structure, the outer problem involves optimizing dual variables derived from the risk measures, while the inner problem involves finding an optimal policy given these dual variables. The proposed method, to the best of our knowledge, is the first to guarantee convergence to an optimum in the tabular setting. Furthermore, the proposed method has been evaluated on continuous control tasks and showed the best performance among other RCRL algorithms satisfying the constraints.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

19 of 32 references.

References · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2017Constrained Policy Optimizationcited by 928
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  6. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  7. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  8. 2020Projection-Based Constrained Policy Optimizationcited by 195
  9. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
  10. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  11. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 60
  12. 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 46
7 more
  1. 2022Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Riskcited by 40
  2. 2022Safe Reinforcement Learning for Legged Locomotioncited by 29
  3. 2022Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 26
  4. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
  5. 2020Minimizing Spectral Risk Measures Applied to Markov Decision Processescited by 12
  6. 2024Safe Reinforcement Learning on the Constraint Manifold: Theory and Applicationscited by 12
  7. 2021Safe Distributional Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learningcited by 6
  2. 2025Optimistic Exploration for Risk-Averse Constrained Reinforcement Learningcited by 3
  3. 2025Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learningcited by 2
  4. 2025Constraint-Aware Reinforcement Learning via Adaptive Action Scalingcited by 0
  5. 2026Utility-Constrained Policy Optimizationcited by 0