mlsanityData to 14 Oct 2026

Burning RED: Unlocking Subtask-Driven Reinforcement Learning and Risk-Awareness in Average-Reward Markov Decision Processes

arXiv 2410.10578 ↗First posted 14 Oct 2024cs.LG cs.AIcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Average-reward Markov decision processes (MDPs) provide a foundational framework for sequential decision-making under uncertainty. However, average-reward MDPs have remained largely unexplored in reinforcement learning (RL) settings, with the majority of RL-based efforts having been allocated to discounted MDPs. In this work, we study a unique structural property of average-reward MDPs and utilize it to introduce Reward-Extended Differential (or RED) reinforcement learning: a novel RL framework that can be used to effectively and efficiently solve various learning objectives, or subtasks, simultaneously in the average-reward setting. We introduce a family of RED learning algorithms for prediction and control, including proven-convergent algorithms for the tabular case. We then showcase the power of these algorithms by demonstrating how they can be used to learn a policy that optimizes, for the first time, the well-known conditional value-at-risk (CVaR) risk measure in a fully-online manner, without the use of an explicit bi-level optimization scheme or an augmented state-space.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 43 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

12 of 43 references.

References · 12

  1. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  2. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  3. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  4. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  5. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 31
  6. 2014Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPscited by 28
  7. 2020Average-reward model-free reinforcement learning: a systematic review and literature mappingcited by 22
  8. 2023On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processescited by 18
  9. 2021Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal controlcited by 12
  10. 2022Continual Auxiliary Task Learningcited by 9
  11. 2022Risk-Sensitive Markov Decision Processes with Long-Run CVaR Criterioncited by 4
  12. 2015Optimal Control of Conditional Value-at-Risk in Continuous Timecited by 3

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025A Differential Perspective on Distributional Reinforcement Learningcited by 5
  2. 2025Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Constrained Reinforcement Learning with Average Reward Objective: Model-Based and Model-Free Algorithms
  2. 2026Average-Reward Reinforcement Learning for Multichain MDPs: A Hierarchical Decomposition Approach
  3. 2024Hierarchical Average-Reward Linearly-solvable Markov Decision Processes
  4. 2024Reinforcement Learning for Infinite-Horizon Average-Reward Linear MDPs via Approximation by Discounted-Reward MDPs
  5. 2025Statistical and Algorithmic Foundations of Reinforcement Learning
  6. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Tasks
  7. 2024Provably Efficient Exploration in Reward Machines with Low Regret
  8. 2025Optimal Single-Policy Sample Complexity and Transient Coverage for Average-Reward Offline RL