mlsanityData to 7 Oct 2026

An Analysis of Quantile Temporal-Difference Learning

arXiv 2301.04462 ↗First posted 11 Jan 2023cs.LG stat.MLcited by 30

Not on the map. Why?

Abstract (arXiv)

We analyse quantile temporal-difference learning (QTD), a distributional reinforcement learning algorithm that has proven to be a key component in several successful large-scale applications of reinforcement learning. Despite these empirical successes, a theoretical understanding of QTD has proven elusive until now. Unlike classical TD learning, which can be analysed with standard stochastic approximation tools, QTD updates do not approximate contraction mappings, are highly non-linear, and may have multiple fixed points. The core result of this paper is a proof of convergence to the fixed points of a related family of dynamic programming procedures with probability 1, putting QTD on firm theoretical footing. The proof establishes connections between QTD and non-linear differential inclusions through stochastic approximation theory and non-smooth analysis.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 11 of its 73 references that are in the map. First posted on arXiv 11 Jan 2023.

Builds on

11 of 73 references.

References · 11

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  3. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  4. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  5. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  6. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
  7. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  8. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  9. 2019Quantile QT-Opt for Risk-Aware Vision-Based Robotic Graspingcited by 27
  10. 2021A Cram'er Distance perspective on Quantile Regression based Distributional Reinforcement Learningcited by 7
  11. 2016Quantile Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 7; 2025: 8; 2026: 3.

30 papers.

Sort:

By area:

  1. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  2. 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learningcited by 23
  3. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 19
  4. 2024Foundations of Multivariate Distributional Reinforcement Learningcited by 16
  5. 2023The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimationcited by 14
  6. 2023Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learningcited by 12
  7. 2023Estimation and Inference in Distributional Reinforcement Learningcited by 8
  8. 2024Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Modelcited by 7
  9. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learningcited by 7
  10. 2024Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysiscited by 6
  11. 2023One-Step Distributional Reinforcement Learningcited by 5
  12. 2023Value-Distributional Model-Based Reinforcement Learningcited by 5
18 more
  1. 2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 5
  2. 2025A Differential Perspective on Distributional Reinforcement Learningcited by 5
  3. 2023Distributional Bellman Operators over Mean Embeddingscited by 4
  4. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximationcited by 3
  5. 2023Variance Control for Distributional Reinforcement Learningcited by 2
  6. 2023The Kernel Density Integral Transformationcited by 2
  7. 2025Survey and Tutorial of Reinforcement Learning Methods in Process Systems Engineeringcited by 2
  8. 2025Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluationcited by 2
  9. 2023Distributional Reinforcement Learning with Dual Expectile-Quantile Regressioncited by 1
  10. 2025ADDQ: Adaptive Distributional Double Q-Learningcited by 1
  11. 2025Pure Exploration via Frank-Wolfe Self-Playcited by 1
  12. 2026Online Inference for Quantile Temporal Difference Learning in Distributional Reinforcement Learningcited by 1
  13. 2023Policy Evaluation in Distributional LQRcited by 0
  14. 2024A Case for Validation Buffer in Pessimistic Actor-Criticcited by 0
  15. 2025Stochastic Approximation Methods for Distortion Risk Measure Optimizationcited by 0
  16. 2025Accelerated Distributional Temporal Difference Learning with Linear Function Approximationcited by 0
  17. 2026Distributional Off-Policy Evaluation with Deep Quantile Process Regressioncited by 0
  18. 2026A Finite-Sample Analysis of Quantile Temporal-Difference Learningcited by 0