# Risk-averse Total-reward MDPs with ERM and EVaR

arXiv: https://arxiv.org/abs/2408.17286 · HTML: https://mlsanity.com/paper/2408.17286/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 47 references that are in the map.
- First posted on arXiv 30 Aug 2024.

## Builds on (most cited first, 13 of 13)

- [Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2111.03947/) (2021)
- [Minimax Regret for Stochastic Shortest Path](https://mlsanity.com/paper/2103.13056/) (2021)
- [Markov Decision Processes with Recursive Risk Measures](https://mlsanity.com/paper/2010.07220/) (2020)
- [On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processes](https://mlsanity.com/paper/2304.12477/) (2023)
- [Constrained Risk-Averse Markov Decision Processes](https://mlsanity.com/paper/2012.02423/) (2020)
- [Beyond Average Return in Markov Decision Processes](https://mlsanity.com/paper/2310.20266/) (2023)
- [Risk Aversion in Finite Markov Decision Processes Using Total Cost Criteria and Average Value at Risk](https://mlsanity.com/paper/1602.05130/) (2016)
- [Distributional Model Equivalence for Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2307.01708/) (2023)
- [Risk-Averse Stochastic Shortest Path Planning](https://mlsanity.com/paper/2103.14727/) (2021)
- [Risk-aware Stochastic Shortest Path](https://mlsanity.com/paper/2203.01640/) (2022)
- [On Exponential Utility and Conditional Value-at-Risk as Risk-Averse Performance Criteria](https://mlsanity.com/paper/2108.01771/) (2021)
- [Solving Multi-Model MDPs by Coordinate Ascent and Dynamic Programming](https://mlsanity.com/paper/2407.06329/) (2024)
- [A Unified Algorithm for Stochastic Path Problems](https://mlsanity.com/paper/2210.09255/) (2022)

## Built on by (most cited first, 4 of 4)

- [Risk-Averse Total-Reward Reinforcement Learning](https://mlsanity.com/paper/2506.21683/) (2025)
- [Boosting CVaR Policy Optimization with Quantile Gradients](https://mlsanity.com/paper/2601.22100/) (2026)
- [Policy Gradient for Continuous-Time Robust Markov Decision Processes](https://mlsanity.com/paper/2606.04335/) (2026)
- [Long-Term Sequential Decision Making under Risk](https://mlsanity.com/paper/2607.19914/) (2026)

## Abstract (arXiv)

Optimizing risk-averse objectives in discounted MDPs is challenging because most models do not admit direct dynamic programming equations and require complex history-dependent policies. In this paper, we show that the risk-averse em total reward criterion, under the Entropic Risk Measure (ERM) and Entropic Value at Risk (EVaR) risk measures, can be optimized by a stationary policy, making it simple to analyze, interpret, and deploy. We propose exponential value iteration, policy iteration, and linear programming to compute optimal policies. Compared with prior work, our results only require the relatively mild condition of transient MDPs and allow for em both positive and negative rewards. Our results indicate that the total reward criterion may be preferable to the discounted criterion in a broad range of risk-averse reinforcement learning domains.