# Risk-Sensitive Markov Decision Processes with Long-Run CVaR Criterion

arXiv: https://arxiv.org/abs/2210.08740 · HTML: https://mlsanity.com/paper/2210.08740/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 47 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 5 of 5)

- [Offline Multi-Action Policy Learning: Generalization and Optimization](https://mlsanity.com/paper/1810.04778/) (2018)
- [On the Complexity of Solving Markov Decision Problems](https://mlsanity.com/paper/1302.4971/) (2013)
- [Risk-Sensitive Markov Decision Processes with Combined Metrics of Mean and Variance](https://mlsanity.com/paper/2008.03707/) (2020)
- [Optimal Control of Conditional Value-at-Risk in Continuous Time](https://mlsanity.com/paper/1512.05015/) (2015)
- [A Generalized Fundamental Matrix for Computing Fundamental Quantities of Markov Systems](https://mlsanity.com/paper/1604.04343/) (2016)

## Built on by (most cited first, 4 of 4)

- [Burning RED: Unlocking Subtask-Driven Reinforcement Learning and Risk-Awareness in Average-Reward Markov Decision Processes](https://mlsanity.com/paper/2410.10578/) (2024)
- [Sharpe Ratio Optimization in Markov Decision Processes](https://mlsanity.com/paper/2509.00793/) (2025)
- [Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learning](https://mlsanity.com/paper/2510.02945/) (2025)
- [Long-Run Conditional Value-at-Risk Reinforcement Learning](https://mlsanity.com/paper/2603.09734/) (2026)

## Abstract (arXiv)

CVaR (Conditional Value at Risk) is a risk metric widely used in finance. However, dynamically optimizing CVaR is difficult since it is not a standard Markov decision process (MDP) and the principle of dynamic programming fails. In this paper, we study the infinite-horizon discrete-time MDP with a long-run CVaR criterion, from the view of sensitivity-based optimization. By introducing a pseudo CVaR metric, we derive a CVaR difference formula which quantifies the difference of long-run CVaR under any two policies. The optimality of deterministic policies is derived. We obtain a so-called Bellman local optimality equation for CVaR, which is a necessary and sufficient condition for local optimal policies and only necessary for global optimal policies. A CVaR derivative formula is also derived for providing more sensitivity information. Then we develop a policy iteration type algorithm to efficiently optimize CVaR, which is shown to converge to local optima in the mixed policy space. We further discuss some extensions including the mean-CVaR optimization and the maximization of CVaR. Finally, we conduct numerical experiments relating to portfolio management to demonstrate the main results. Our work may shed light on dynamically optimizing CVaR from a sensitivity viewpoint.