mlsanityData to 7 Oct 2026

Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis

arXiv 2410.24128 ↗First posted 31 Oct 2024cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In Markov decision processes (MDPs), quantile risk measures such as Value-at-Risk are a standard metric for modeling RL agents' preferences for certain outcomes. This paper proposes a new Q-learning algorithm for quantile optimization in MDPs with strong convergence and performance guarantees. The algorithm leverages a new, simple dynamic program (DP) decomposition for quantile MDPs. Compared with prior work, our DP decomposition requires neither known transition probabilities nor solving complex saddle point equations and serves as a suitable foundation for other model-free RL algorithms. Our numerical results in tabular domains show that our Q-learning algorithm converges to its DP variant and outperforms earlier algorithms.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 19 of its 52 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

19 of 52 references.

References · 19

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  3. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  4. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  5. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  6. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  7. 2013Risk-sensitive Reinforcement Learningcited by 129
  8. 2014Algorithms for CVaR Optimization in MDPscited by 98
  9. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  10. 2017How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Roboticscited by 57
  11. 2020Partial Policy Iteration for L1-Robust Markov Decision Processescited by 49
  12. 2023An Analysis of Quantile Temporal-Difference Learningcited by 30
7 more
  1. 2023On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processescited by 17
  2. 2021Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal controlcited by 12
  3. 2023An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradientcited by 12
  4. 2020Wasserstein Distributionally Robust Motion Control for Collision Avoidance Using Conditional Value-at-Riskcited by 8
  5. 2021Prospect-theoretic Q-learningcited by 6
  6. 2016Quantile Reinforcement Learningcited by 2
  7. 2022Risk-Sensitive Optimal Execution via a Conditional Value-at-Risk Objectivecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Planning and Learning in Average Risk-aware MDPscited by 2
  2. 2025Risk-Averse Total-Reward Reinforcement Learningcited by 0
  3. 2026Boosting CVaR Policy Optimization with Quantile Gradientscited by 0
  4. 2026Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinitycited by 0
  5. 2026Stochastic Minimum-Cost Reach-Avoid Reinforcement Learningcited by 0
  6. 2026Online Policy Evaluation for MDPs with Dynamic UBSR Measurescited by 0