mlsanityData to 7 Oct 2026

Risk-averse Total-reward MDPs with ERM and EVaR

arXiv 2408.17286 ↗First posted 30 Aug 2024cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Optimizing risk-averse objectives in discounted MDPs is challenging because most models do not admit direct dynamic programming equations and require complex history-dependent policies. In this paper, we show that the risk-averse em total reward criterion, under the Entropic Risk Measure (ERM) and Entropic Value at Risk (EVaR) risk measures, can be optimized by a stationary policy, making it simple to analyze, interpret, and deploy. We propose exponential value iteration, policy iteration, and linear programming to compute optimal policies. Compared with prior work, our results only require the relatively mild condition of transient MDPs and allow for em both positive and negative rewards. Our results indicate that the total reward criterion may be preferable to the discounted criterion in a broad range of risk-averse reinforcement learning domains.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 13 of its 47 references that are in the map. First posted on arXiv 30 Aug 2024.

Builds on

13 of 47 references.

References · 13

  1. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 42
  2. 2021Minimax Regret for Stochastic Shortest Pathcited by 30
  3. 2020Markov Decision Processes with Recursive Risk Measurescited by 17
  4. 2023On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processescited by 17
  5. 2020Constrained Risk-Averse Markov Decision Processescited by 15
  6. 2023Beyond Average Return in Markov Decision Processescited by 11
  7. 2023Distributional Model Equivalence for Risk-Sensitive Reinforcement Learningcited by 9
  8. 2016Risk Aversion in Finite Markov Decision Processes Using Total Cost Criteria and Average Value at Riskcited by 8
  9. 2021Risk-Averse Stochastic Shortest Path Planningcited by 8
  10. 2022Risk-aware Stochastic Shortest Pathcited by 4
  11. 2021On Exponential Utility and Conditional Value-at-Risk as Risk-Averse Performance Criteriacited by 3
  12. 2024Solving Multi-Model MDPs by Coordinate Ascent and Dynamic Programmingcited by 2
1 more
  1. 2022A Unified Algorithm for Stochastic Path Problemscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Risk-Averse Total-Reward Reinforcement Learningcited by 0
  2. 2026Boosting CVaR Policy Optimization with Quantile Gradientscited by 0
  3. 2026Policy Gradient for Continuous-Time Robust Markov Decision Processescited by 0
  4. 2026Long-Term Sequential Decision Making under Riskcited by 0