mlsanityData to 7 Oct 2026

The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation

arXiv 2305.18388 ↗First posted 28 May 2023cs.LG stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

We study the problem of temporal-difference-based policy evaluation in reinforcement learning. In particular, we analyse the use of a distributional reinforcement learning algorithm, quantile temporal-difference learning (QTD), for this task. We reach the surprising conclusion that even if a practitioner has no interest in the return distribution beyond the mean, QTD (which learns predictions about the full distribution of returns) may offer performance superior to approaches such as classical TD learning, which predict only the mean return, even in the tabular setting.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 30 of its 79 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

30 of 79 references.

References · 30

  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  3. 2023Mastering Diverse Domains through World Modelscited by 1,193
  4. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  6. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  7. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  8. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  9. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
  10. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  11. 2020Munchausen Reinforcement Learningcited by 68
  12. 2018Improving Regression Performance with Distributional Lossescited by 66
18 more
  1. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  2. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  3. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  4. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  5. 2020Gradient Temporal-Difference Learning with Regularized Correctionscited by 38
  6. 2023An Analysis of Quantile Temporal-Difference Learningcited by 30
  7. 2019Quantile QT-Opt for Risk-Aware Vision-Based Robotic Graspingcited by 27
  8. 2016A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learningcited by 20
  9. 2017Multi-step Off-policy Learning Without Importance Sampling Ratioscited by 19
  10. 2019Adaptive Trade-Offs in Off-Policy Learningcited by 19
  11. 2020Regularized Off-Policy TD-Learningcited by 10
  12. 2023On the Statistical Benefits of Temporal Difference Learningcited by 8
  13. 2019Robust Temporal Difference Learning for Critical Domainscited by 7
  14. 2021A Cram'er Distance perspective on Quantile Regression based Distributional Reinforcement Learningcited by 7
  15. 2021Robustness and risk management via distributional dynamic programmingcited by 5
  16. 2023One-Step Distributional Reinforcement Learningcited by 5
  17. 2021Robust and Adaptive Temporal-Difference Learning Using An Ensemble of Gaussian Processescited by 4
  18. 2021Distributional Reinforcement Learning with Unconstrained Monotonic Neural Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 4; 2026: 5.

14 papers.

Sort:

By area:

  1. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  2. 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learningcited by 23
  3. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
  4. 2023Beyond Average Return in Markov Decision Processescited by 11
  5. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
  6. 2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 5
  7. 2023Variance Control for Distributional Reinforcement Learningcited by 2
  8. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  9. 2025ADDQ: Adaptive Distributional Double Q-Learningcited by 1
  10. 2026Online Inference for Quantile Temporal Difference Learning in Distributional Reinforcement Learningcited by 1
  11. 2026Distributional Off-Policy Evaluation with Deep Quantile Process Regressioncited by 0
  12. 2026Auditing the Risk Claims of Distributional Reinforcement Learningcited by 0
2 more
  1. 2026Revisiting TD Target Aggregation under Uncertainty in Q-Learningcited by 0
  2. 2026A Finite-Sample Analysis of Quantile Temporal-Difference Learningcited by 0