mlsanityData to 14 Oct 2026

Measures of Variability for Risk-averse Policy Gradient

arXiv 2504.11412 ↗First posted 15 Apr 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Risk-averse reinforcement learning (RARL) is critical for decision-making under uncertainty, which is especially valuable in high-stake applications. However, most existing works focus on risk measures, e.g., conditional value-at-risk (CVaR), while measures of variability remain underexplored. In this paper, we comprehensively study nine common measures of variability, namely Variance, Gini Deviation, Mean Deviation, Mean-Median Deviation, Standard Deviation, Inter-Quantile Range, CVaR Deviation, Semi_Variance, and Semi_Standard Deviation. Among them, four metrics have not been previously studied in RARL. We derive policy gradient formulas for these unstudied metrics, improve gradient estimation for Gini Deviation, analyze their gradient properties, and incorporate them with the REINFORCE and PPO frameworks to penalize the dispersion of returns. Our empirical study reveals that variance-based metrics lead to unstable policy updates. In contrast, CVaR Deviation and Gini Deviation show consistent performance across different randomness and evaluation domains, achieving high returns while effectively learning risk-averse policies. Mean Deviation and Semi_Standard Deviation are also competitive across different scenarios. This work provides a comprehensive overview of variability measures in RARL, offering practical insights for risk-aware decision-making and guiding future research on risk metrics and RARL algorithms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 60 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

20 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization17 Mar

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  4. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  5. 2014Optimizing the CVaR via Samplingcited by 99
  6. 2014Algorithms for CVaR Optimization in MDPscited by 99
  7. 2015Policy Gradient for Coherent Risk Measurescited by 76
  8. 2019Risk-Averse Trust Region Optimization for Reward-Volatility Reductioncited by 54
  9. 2022Efficient Risk-Averse Reinforcement Learningcited by 43
  10. 2022Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Riskcited by 41
  11. 2020Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learningcited by 28
  12. 2022Benchmarking Constraint Inference in Inverse Reinforcement Learningcited by 18
7 more
  1. 2023An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradientcited by 12
  2. 2018A Block Coordinate Ascent Algorithm for Mean-Variance Optimizationcited by 11
  3. 2022Quantile-Based Policy Optimization for Reinforcement Learningcited by 8
  4. 2023TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learningcited by 5
  5. 2018Risk-averse Behavior Planning for Autonomous Driving under Uncertaintycited by 4
  6. 2023Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithmscited by 4
  7. 2022Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning
  2. 2026Boosting CVaR Policy Optimization with Quantile Gradients
  3. 2024On the Approximation and Convergence of Distributional Policy Gradient Algorithms for Risk-Sensitive Reinforcement Learning
  4. 2026Long-Run Conditional Value-at-Risk Reinforcement Learning
  5. 2026Cantelli Constrained Policy Optimization
  6. 2025Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning
  7. 2024Robust Reinforcement Learning with Dynamic Distortion Risk Measures
  8. 2025Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning