# Planning for Risk-Aversion and Expected Value in MDPs

arXiv: https://arxiv.org/abs/2110.12746 · HTML: https://mlsanity.com/paper/2110.12746/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 42 references that are in the map.
- First posted on arXiv 25 Oct 2021.

## Builds on (most cited first, 10 of 10)

- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach](https://mlsanity.com/paper/1506.02188/) (2015)
- [Optimizing the CVaR via Sampling](https://mlsanity.com/paper/1404.3862/) (2014)
- [Algorithms for CVaR Optimization in MDPs](https://mlsanity.com/paper/1406.3339/) (2014)
- [Policy Gradient for Coherent Risk Measures](https://mlsanity.com/paper/1502.03919/) (2015)
- [Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy](https://mlsanity.com/paper/1911.01546/) (2019)
- [How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics](https://mlsanity.com/paper/1710.11040/) (2017)
- [Worst Cases Policy Gradients](https://mlsanity.com/paper/1911.03618/) (2019)
- [Risk-Averse Bayes-Adaptive Reinforcement Learning](https://mlsanity.com/paper/2102.05762/) (2021)
- [Policy Gradients for CVaR-Constrained MDPs](https://mlsanity.com/paper/1405.2690/) (2014)

## Built on by (most cited first, 5 of 5)

- [One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2212.00124/) (2022)
- [Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief](https://mlsanity.com/paper/2210.06692/) (2022)
- [Offline Policy Optimization with Posterior Sampling](https://mlsanity.com/paper/2605.07393/) (2026)
- [Forward-PECVaR Algorithm: Exact Evaluation for CVaR SSPs](https://mlsanity.com/paper/2303.00672/) (2023)
- [Long-Term Sequential Decision Making under Risk](https://mlsanity.com/paper/2607.19914/) (2026)

## Abstract (arXiv)

Planning in Markov decision processes (MDPs) typically optimises the expected cost. However, optimising the expectation does not consider the risk that for any given run of the MDP, the total cost received may be unacceptably high. An alternative approach is to find a policy which optimises a risk-averse objective such as conditional value at risk (CVaR). However, optimising the CVaR alone may result in poor performance in expectation. In this work, we begin by showing that there can be multiple policies which obtain the optimal CVaR. This motivates us to propose a lexicographic approach which minimises the expected cost subject to the constraint that the CVaR of the total cost is optimal. We present an algorithm for this problem and evaluate our approach on four domains. Our results demonstrate that our lexicographic approach improves the expected cost compared to the state of the art algorithm, while achieving the optimal CVaR.