# Measures of Variability for Risk-averse Policy Gradient

arXiv: https://arxiv.org/abs/2504.11412 · HTML: https://mlsanity.com/paper/2504.11412/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 20 of its 60 references that are in the map.
- First posted on arXiv 15 Apr 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach](https://mlsanity.com/paper/1506.02188/) (2015)
- [Optimizing the CVaR via Sampling](https://mlsanity.com/paper/1404.3862/) (2014)
- [Algorithms for CVaR Optimization in MDPs](https://mlsanity.com/paper/1406.3339/) (2014)
- [Policy Gradient for Coherent Risk Measures](https://mlsanity.com/paper/1502.03919/) (2015)
- [Risk-Averse Trust Region Optimization for Reward-Volatility Reduction](https://mlsanity.com/paper/1912.03193/) (2019)
- [Efficient Risk-Averse Reinforcement Learning](https://mlsanity.com/paper/2205.05138/) (2022)
- [Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk](https://mlsanity.com/paper/2206.04436/) (2022)
- [Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning](https://mlsanity.com/paper/2004.10888/) (2020)
- [Benchmarking Constraint Inference in Inverse Reinforcement Learning](https://mlsanity.com/paper/2206.09670/) (2022)
- [An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient](https://mlsanity.com/paper/2307.08873/) (2023)
- [A Block Coordinate Ascent Algorithm for Mean-Variance Optimization](https://mlsanity.com/paper/1809.02292/) (2018)
- [Quantile-Based Policy Optimization for Reinforcement Learning](https://mlsanity.com/paper/2201.11463/) (2022)
- [A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization](https://mlsanity.com/paper/2403.11062/) (2024)
- [TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learning](https://mlsanity.com/paper/2312.00344/) (2023)
- [Risk-averse Behavior Planning for Autonomous Driving under Uncertainty](https://mlsanity.com/paper/1812.01254/) (2018)
- [Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithms](https://mlsanity.com/paper/2305.07248/) (2023)
- [Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning](https://mlsanity.com/paper/2206.07376/) (2022)

## Abstract (arXiv)

Risk-averse reinforcement learning (RARL) is critical for decision-making under uncertainty, which is especially valuable in high-stake applications. However, most existing works focus on risk measures, e.g., conditional value-at-risk (CVaR), while measures of variability remain underexplored. In this paper, we comprehensively study nine common measures of variability, namely Variance, Gini Deviation, Mean Deviation, Mean-Median Deviation, Standard Deviation, Inter-Quantile Range, CVaR Deviation, Semi_Variance, and Semi_Standard Deviation. Among them, four metrics have not been previously studied in RARL. We derive policy gradient formulas for these unstudied metrics, improve gradient estimation for Gini Deviation, analyze their gradient properties, and incorporate them with the REINFORCE and PPO frameworks to penalize the dispersion of returns. Our empirical study reveals that variance-based metrics lead to unstable policy updates. In contrast, CVaR Deviation and Gini Deviation show consistent performance across different randomness and evaluation domains, achieving high returns while effectively learning risk-averse policies. Mean Deviation and Semi_Standard Deviation are also competitive across different scenarios. This work provides a comprehensive overview of variability measures in RARL, offering practical insights for risk-aware decision-making and guiding future research on risk metrics and RARL algorithms.