mlsanityData to 14 Oct 2026

Planning and Learning in Average Risk-aware MDPs

arXiv 2503.17629 ↗First posted 22 Mar 2025cs.LG math.OCcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

For continuing tasks, average cost Markov decision processes have well-documented value and can be solved using efficient algorithms. However, it explicitly assumes that the agent is risk-neutral. In this work, we extend risk-neutral algorithms to accommodate the more general class of dynamic risk measures. Specifically, we propose a relative value iteration (RVI) algorithm for planning and design two model-free Q-learning algorithms, namely a generic algorithm based on the multi-level Monte Carlo (MLMC) method, and an off-policy algorithm dedicated to utility-based shortfall risk measures. Both the RVI and MLMC-based Q-learning algorithms are proven to converge to optimality. Numerical experiments validate our analysis, confirm empirically the convergence of the off-policy algorithm, and demonstrate that our approach enables the identification of policies that are finely tuned to the intricate risk-awareness of the agent that they serve.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 52 references that are in the map. First posted on arXiv 22 Mar 2025.

Builds on

19 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis31 Oct
  2. 2024On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processes29 Aug

Other references · 17

  1. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  2. 2013Risk-sensitive Reinforcement Learningcited by 130
  3. 2014Optimizing the CVaR via Samplingcited by 99
  4. 2014Algorithms for CVaR Optimization in MDPscited by 99
  5. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  6. 2023A Finite Sample Complexity Bound for Distributionally Robust Q-learningcited by 36
  7. 2023Sample Complexity of Variance-reduced Distributionally Robust Q-learningcited by 29
  8. 2019Discounted Reinforcement Learning Is Not an Optimization Problemcited by 26
  9. 2023Robust Average-Reward Markov Decision Processescited by 22
  10. 2023Model-Free Robust Average-Reward Reinforcement Learningcited by 20
  11. 2019A Wasserstein distance approach for concentration of empirical risk estimatescited by 18
  12. 2019Unbiased Multilevel Monte Carlo: Stochastic Optimization, Steady-state Simulation, Quantiles, and Other Applicationscited by 18
5 more
  1. 2020Risk-Averse Learning by Temporal Difference Methodscited by 13
  2. 2022A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDPcited by 13
  3. 2022Ergodic Risk-sensitive control -- A surveycited by 4
  4. 2014On Average Risk-sensitive Markov Control Processescited by 1
  5. 2019On the relative value iteration with a risk-sensitive criterioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learningcited by 0
  2. 2026Online Policy Evaluation for MDPs with Dynamic UBSR Measurescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Bayesian Risk-Sensitive Policy Optimization For MDPs With General Loss Functions
  2. 2025Risk-Averse Total-Reward Reinforcement Learning
  3. 2026Risk-Aware General-Utility Markov Decision Processes
  4. 2021An Offline Risk-aware Policy Selection Method for Bayesian Markov Decision Processes
  5. 2025Efficient Algorithms for Mitigating Uncertainty and Risk in Reinforcement Learning
  6. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning
  7. 2025Recursive Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Model
  8. 2025An Actor-Critic Algorithm with Function Approximation for Risk Sensitive Cost Markov Decision Processes