mlsanityData to 14 Oct 2026

Risk-Averse Total-Reward Reinforcement Learning

arXiv 2506.21683 ↗First posted 26 Jun 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Risk-averse total-reward Markov Decision Processes (MDPs) offer a promising framework for modeling and solving undiscounted infinite-horizon objectives. Existing model-based algorithms for risk measures like the entropic risk measure (ERM) and entropic value-at-risk (EVaR) are effective in small problems, but require full access to transition probabilities. We propose a Q-learning algorithm to compute the optimal stationary policy for total-reward ERM and EVaR objectives with strong convergence and performance guarantees. The algorithm and its optimality are made possible by ERM's dynamic consistency and elicitability. Our numerical results on tabular domains demonstrate quick and reliable convergence of the proposed Q-learning algorithm to the optimal risk-averse value function.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 47 references that are in the map. First posted on arXiv 26 Jun 2025.

Builds on

20 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis31 Oct

Other references · 19

  1. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  2. 2013Risk-sensitive Reinforcement Learningcited by 130
  3. 2021Risk-Averse Offline Reinforcement Learningcited by 65
  4. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  5. 2019Risk Averse Robust Adversarial Reinforcement Learningcited by 57
  6. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 43
  7. 2022Efficient Risk-Averse Reinforcement Learningcited by 43
  8. 2021Minimax Regret for Stochastic Shortest Pathcited by 30
  9. 2020Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learningcited by 28
  10. 2020Markov Decision Processes with Recursive Risk Measurescited by 18
  11. 2023On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processescited by 18
  12. 2023TD Convergence: An Optimization Perspectivecited by 16
7 more
  1. 2023Beyond Average Return in Markov Decision Processescited by 11
  2. 2023Distributional Model Equivalence for Risk-Sensitive Reinforcement Learningcited by 9
  3. 2021Risk-Averse Stochastic Shortest Path Planningcited by 8
  4. 2022Risk-aware Stochastic Shortest Pathcited by 4
  5. 2024Risk-averse Total-reward MDPs with ERM and EVaRcited by 4
  6. 2024Solving Multi-Model MDPs by Coordinate Ascent and Dynamic Programmingcited by 2
  7. 2021A Risk-Averse Preview-based Q-Learning Algorithm: Application to Highway Driving of Autonomous Vehiclescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Algorithms for Mitigating Uncertainty and Risk in Reinforcement Learning
  2. 2025Planning and Learning in Average Risk-aware MDPs
  3. 2024Reinforcement Learning for Infinite-Horizon Average-Reward Linear MDPs via Approximation by Discounted-Reward MDPs
  4. 2026Reinforcement Learning for Exponential Utility: Algorithms and Convergence in Discounted MDPs
  5. 2025Recursive Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Model
  6. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning
  7. 2025Convergent Q-Learning for Infinite-Horizon General-Sum Markov Games through Behavioral Economics
  8. 2025Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning