Data to 7 Oct 2026

Algorithms for CVaR Optimization in MDPs

arXiv 1406.3339 ↗First posted 12 Jun 2014cs.AI math.OCcited by 98

Not on the map. Why?

Abstract (arXiv)

In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in costs in addition to minimizing a standard criterion. Conditional value-at-risk (CVaR) is a relatively new risk measure that addresses some of the shortcomings of the well-known variance-related risk measures, and because of its computational efficiencies has gained popularity in finance and operations research. In this paper, we consider the mean-CVaR optimization problem in MDPs. We first derive a formula for computing the gradient of this risk-sensitive objective function. We then devise policy gradient and actor-critic algorithms that each uses a specific method to estimate this gradient and updates the policy parameters in the descent direction. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in an optimal stopping problem.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jun 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 17; 2022: 18; 2023: 18; 2024: 10; 2025: 13; 2026: 21.

98 papers.

Sort:

By area:

2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
2021Risk-Averse Offline Reinforcement Learningcited by 65
2022Efficient Risk-Averse Reinforcement Learningcited by 41
2022Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Riskcited by 40
2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
2021RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agentscited by 31
2021Risk-Averse Bayes-Adaptive Reinforcement Learningcited by 28
2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 27
2022Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 26
2022Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guaranteescited by 24
2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 21
2021On the Convergence and Optimality of Policy Gradient for Markov Coherent Riskcited by 19

38 more

2023RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorizationcited by 19
2021Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Drivingcited by 17
2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
2023On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processescited by 17
2018Risk-Sensitive Generative Adversarial Imitation Learningcited by 16
2021SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learningcited by 15
2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 15
2022Deep Hedging: Continuous Reinforcement Learning for Hedging of General Portfolios across Multiple Risk Aversionscited by 13
2021Lyapunov Barrier Policy Optimizationcited by 12
2022RAP: Risk-Aware Prediction for Robust Planningcited by 12
2023An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradientcited by 12
2021Variance Penalized On-Policy and Off-Policy Actor-Criticcited by 10
2021Safe Distributional Reinforcement Learningcited by 10
2024RACER: Epistemic Risk-Sensitive RL Enables Fast Driving with Fewer Crashescited by 10
2022Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learningcited by 9
2023Distributional Model Equivalence for Risk-Sensitive Reinforcement Learningcited by 9
2021Risk-Averse Stochastic Shortest Path Planningcited by 8
2023Risk-Aware Reinforcement Learning through Optimal Transport Theorycited by 7
2022SAAC: Safe Reinforcement Learning as an Adversarial Game of Actor-Criticscited by 6
2023Learning Diverse Risk Preferences in Population-based Self-playcited by 6
2023Provably Efficient CVaR RL in Low-rank MDPscited by 6
2024Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shiftscited by 6
2024Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysiscited by 6
2025Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learningcited by 6
2021Automatic Risk Adaptation in Distributional Reinforcement Learningcited by 5
2021Planning for Risk-Aversion and Expected Value in MDPscited by 5
2023A Model-Based Method for Minimizing CVaR and Beyondcited by 5
2023Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theorycited by 5
2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 5
2022Risk-Driven Design of Perception Systemscited by 4
2022RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Riskcited by 4
2022Non-stationary Risk-sensitive Reinforcement Learning: Near-optimal Dynamic Regret, Adaptive Detection, and Separation Designcited by 4
2023Risk-Averse Reinforcement Learning via Dynamic Time-Consistent Risk Measurescited by 4
2023TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learningcited by 4
2023Noise Distribution Decomposition based Multi-Agent Distributional Reinforcement Learningcited by 4
2025Efficient Risk-sensitive Planning via Entropic Risk Measurescited by 4
2026Insuring Every Action: An Authority Frontier Framework for Runtime Actuarial Control of Autonomous AI Agentscited by 4

50 most cited of 98 All 98 (JSON)