mlsanityData to 14 Oct 2026

A Differential Perspective on Distributional Reinforcement Learning

arXiv 2506.03333 ↗First posted 3 Jun 2025cs.LG cs.AIcited by 5

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

To date, distributional reinforcement learning (distributional RL) methods have exclusively focused on the discounted setting, where an agent aims to optimize a discounted sum of rewards over time. In this work, we extend distributional RL to the average-reward setting, where an agent aims to optimize the reward received per time step. In particular, we utilize a quantile-based approach to develop the first set of algorithms that can successfully learn and/or optimize the long-run per-step reward distribution, as well as the differential return distribution of an average-reward MDP. We derive proven-convergent tabular algorithms for both prediction and control, as well as a broader family of algorithms that have appealing scaling properties. Empirically, we find that these algorithms yield competitive and sometimes superior performance when compared to their non-distributional equivalents, while also capturing rich information about the long-run per-step reward and differential return distributions.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 8 of its 24 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

8 of 24 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Burning RED: Unlocking Subtask-Driven Reinforcement Learning and Risk-Awareness in Average-Reward Markov Decision Processes14 Oct

Other references · 7

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  3. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  4. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  5. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  6. 2023An Analysis of Quantile Temporal-Difference Learningcited by 31
  7. 2019Discounted Reinforcement Learning Is Not an Optimization Problemcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026Maximum Entropy Exploration Without the Rolloutscited by 1
  2. 2026Quotient-Categorical Representations for Bellman-Compatible Average-Reward Distributional Reinforcement Learningcited by 1
  3. 2025Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learningcited by 0
  4. 2026Thermodynamics of Reinforcement Learning Curriculacited by 0
  5. 2026Deep Reinforcement Learning and The Tale of Two Temporal Difference Errorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On Policy Evaluation Algorithms in Distributional Reinforcement Learning
  2. 2026Statistical Efficiency and Inference of Quantile Distributional Reinforcement Learning
  3. 2024Foundations of Multivariate Distributional Reinforcement Learning
  4. 2025Optimizing Return Distributions with Distributional Dynamic Programming
  5. 2024A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learning
  6. 2024The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards
  7. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning
  8. 2024Statistical Efficiency of Distributional Temporal Difference Learning and Freedman's Inequality in Hilbert Spaces