# Policy Gradients for CVaR-Constrained MDPs

arXiv: https://arxiv.org/abs/1405.2690 · HTML: https://mlsanity.com/paper/1405.2690/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2014.

## Built on by (most cited first, 21 of 21)

- [Risk-Averse Bayes-Adaptive Reinforcement Learning](https://mlsanity.com/paper/2102.05762/) (2021)
- [Risk-Sensitive Reinforcement Learning with Exponential Criteria](https://mlsanity.com/paper/2212.09010/) (2022)
- [A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safety](https://mlsanity.com/paper/2505.17342/) (2025)
- [Risk-Averse Stochastic Shortest Path Planning](https://mlsanity.com/paper/2103.14727/) (2021)
- [Policy Gradient Methods for Distortion Risk Measures](https://mlsanity.com/paper/2107.04422/) (2021)
- [Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts](https://mlsanity.com/paper/2402.09992/) (2024)
- [Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning](https://mlsanity.com/paper/2407.07631/) (2024)
- [Planning for Risk-Aversion and Expected Value in MDPs](https://mlsanity.com/paper/2110.12746/) (2021)
- [Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Intervention](https://mlsanity.com/paper/2111.05819/) (2021)
- [Learning to Act Safely with Limited Exposure and Almost Sure Certainty](https://mlsanity.com/paper/2105.08748/) (2021)
- [Risk-Averse Decision Making Under Uncertainty](https://mlsanity.com/paper/2109.04082/) (2021)
- [A policy gradient approach for optimization of smooth risk measures](https://mlsanity.com/paper/2202.11046/) (2022)
- [Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Risk](https://mlsanity.com/paper/2405.01718/) (2024)
- [A Risk-Averse Preview-based Q-Learning Algorithm: Application to Highway Driving of Autonomous Vehicles](https://mlsanity.com/paper/2112.03232/) (2021)
- [Reinforcement Learning for Exponential Utility: Algorithms and Convergence in Discounted MDPs](https://mlsanity.com/paper/2605.08053/) (2026)
- [Forward-PECVaR Algorithm: Exact Evaluation for CVaR SSPs](https://mlsanity.com/paper/2303.00672/) (2023)
- [Percentile Criterion Optimization in Offline Reinforcement Learning](https://mlsanity.com/paper/2404.05055/) (2024)
- [Long-Run Conditional Value-at-Risk Reinforcement Learning](https://mlsanity.com/paper/2603.09734/) (2026)
- [Adaptive Finite-Budget Training for CVaR Risk-Aware Q-Learning](https://mlsanity.com/paper/2608.04305/) (2026)
- [Proactive Context-Forecasted Safety Constraints for Nonstationary Reinforcement Learning](https://mlsanity.com/paper/2609.08080/) (2026)
- [Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedback](https://mlsanity.com/paper/2609.38938/) (2026)

## Abstract (arXiv)

We study a risk-constrained version of the stochastic shortest path (SSP) problem, where the risk measure considered is Conditional Value-at-Risk (CVaR). We propose two algorithms that obtain a locally risk-optimal policy by employing four tools: stochastic approximation, mini batches, policy gradients and importance sampling. Both the algorithms incorporate a CVaR estimation procedure, along the lines of Bardou et al. [2009], which in turn is based on Rockafellar-Uryasev's representation for CVaR and utilize the likelihood ratio principle for estimating the gradient of the sum of one cost function (objective of the SSP) and the gradient of the CVaR of the sum of another cost function (in the constraint of SSP). The algorithms differ in the manner in which they approximate the CVaR estimates/necessary gradients - the first algorithm uses stochastic approximation, while the second employ mini-batches in the spirit of Monte Carlo methods. We establish asymptotic convergence of both the algorithms. Further, since estimating CVaR is related to rare-event simulation, we incorporate an importance sampling based variance reduction scheme into our proposed algorithms.