# A policy gradient approach for optimization of smooth risk measures

arXiv: https://arxiv.org/abs/2202.11046 · HTML: https://mlsanity.com/paper/2202.11046/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 37 references that are in the map.
- First posted on arXiv 22 Feb 2022.

## Builds on (most cited first, 11 of 11)

- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies](https://mlsanity.com/paper/1906.08383/) (2019)
- [An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedback](https://mlsanity.com/paper/1507.08752/) (2015)
- [Stochastic Variance-Reduced Policy Gradient](https://mlsanity.com/paper/1806.05618/) (2018)
- [Policy Gradient for Coherent Risk Measures](https://mlsanity.com/paper/1502.03919/) (2015)
- [Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Control](https://mlsanity.com/paper/1506.02632/) (2015)
- [Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPs](https://mlsanity.com/paper/1403.6530/) (2014)
- [Policy Gradients for CVaR-Constrained MDPs](https://mlsanity.com/paper/1405.2690/) (2014)
- [Spectral risk-based learning using unbounded losses](https://mlsanity.com/paper/2105.04816/) (2021)
- [Policy Gradient Methods for Distortion Risk Measures](https://mlsanity.com/paper/2107.04422/) (2021)
- [Smoothed functional-based gradient algorithms for off-policy reinforcement learning: A non-asymptotic viewpoint](https://mlsanity.com/paper/2101.02137/) (2021)

## Built on by (most cited first, 2 of 2)

- [Catastrophic-risk-aware reinforcement learning with extreme-value-theory-based policy gradients](https://mlsanity.com/paper/2406.15612/) (2024)
- [Percentile Criterion Optimization in Offline Reinforcement Learning](https://mlsanity.com/paper/2404.05055/) (2024)

## Abstract (arXiv)

We propose policy gradient algorithms for solving a risk-sensitive reinforcement learning (RL) problem in on-policy as well as off-policy settings. We consider episodic Markov decision processes, and model the risk using the broad class of smooth risk measures of the cumulative discounted reward. We propose two template policy gradient algorithms that optimize a smooth risk measure in on-policy and off-policy RL settings, respectively. We derive non-asymptotic bounds that quantify the rate of convergence of our proposed algorithms to a stationary point of the smooth risk measure. As special cases, we establish that our algorithms apply to optimization of mean-variance and distortion risk measures, respectively.