mlsanityData to 13 Oct 2026

Beyond Average Return in Markov Decision Processes

arXiv 2310.20266 ↗First posted 31 Oct 2023cs.AI math.OC math.PRcited by 11

Not on the map. Why?

Abstract (arXiv)

What are the functionals of the reward that can be computed and optimized exactly in Markov Decision Processes?In the finite-horizon, undiscounted setting, Dynamic Programming (DP) can only handle these operations efficiently for certain classes of statistics. We summarize the characterization of these classes for policy evaluation, and give a new answer for the planning problem. Interestingly, we prove that only generalized means can be optimized exactly, even in the more general framework of Distributional Reinforcement Learning (DistRL).DistRL permits, however, to evaluate other functionals approximately. We provide error bounds on the resulting estimators, and discuss the potential of this approach as well as its limitations.These results contribute to advancing the theory of Markov Decision Processes by examining overall characteristics of the return, and particularly risk-conscious strategies.

Key facts. Cited by 11 papers in the map (data through 13 Oct 2026). Builds on 18 of its 40 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

18 of 40 references.

References · 18

  1. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 777
  2. 2017A Distributional Perspective on Reinforcement Learningcited by 745
  3. 2018Is Q-learning Provably Efficient?cited by 485
  4. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 465
  5. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 363
  6. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  7. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  8. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  9. 2017Deep Reinforcement Learning for De-Novo Drug Designcited by 161
  10. 2013Risk-sensitive Reinforcement Learningcited by 129
  11. 2020Chip Placement with Deep Reinforcement Learningcited by 83
  12. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
6 more
  1. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  2. 2022Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Boundscited by 16
  3. 2023The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimationcited by 14
  4. 2021Addressing Inherent Uncertainty: Risk-Sensitive Behavior Generation for Automated Driving using Distributional Reinforcement Learningcited by 13
  5. 2021Robustness and risk management via distributional dynamic programmingcited by 5
  6. 2022Distributional Reinforcement Learning for Scheduling of Chemical Production Processescited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 5; 2026: 2.

11 papers.

  1. 2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 6
  2. 2025Efficient Risk-sensitive Planning via Entropic Risk Measurescited by 5
  3. 2023Distributional Bellman Operators over Mean Embeddingscited by 4
  4. 2024Risk-averse Total-reward MDPs with ERM and EVaRcited by 4
  5. 2025Recursive Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Modelcited by 4
  6. 2026On the Sample Complexity of Discounted Reinforcement Learning with Optimized Certainty Equivalentscited by 1
  7. 2024Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximationcited by 0
  8. 2024Variational Bayes Portfolio Constructioncited by 0
  9. 2025Risk-Averse Total-Reward Reinforcement Learningcited by 0
  10. 2025EVaR-Optimal Arm Identification in Banditscited by 0
  11. 2026Utility-Constrained Policy Optimizationcited by 0