# An Analysis of Quantile Temporal-Difference Learning

arXiv: https://arxiv.org/abs/2301.04462 · HTML: https://mlsanity.com/paper/2301.04462/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 73 references that are in the map.
- First posted on arXiv 11 Jan 2023.

## Builds on (most cited first, 11 of 11)

- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [Fully Parameterized Quantile Function for Distributional Reinforcement Learning](https://mlsanity.com/paper/1911.02140/) (2019)
- [An Analysis of Categorical Distributional Reinforcement Learning](https://mlsanity.com/paper/1802.08163/) (2018)
- [Statistics and Samples in Distributional Reinforcement Learning](https://mlsanity.com/paper/1902.08102/) (2019)
- [Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping](https://mlsanity.com/paper/1910.02787/) (2019)
- [A Cram'er Distance perspective on Quantile Regression based Distributional Reinforcement Learning](https://mlsanity.com/paper/2110.00535/) (2021)
- [Quantile Reinforcement Learning](https://mlsanity.com/paper/1611.00862/) (2016)

## Built on by (most cited first, 30 of 31)

- [The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learning](https://mlsanity.com/paper/2305.15703/) (2023)
- [More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning](https://mlsanity.com/paper/2402.07198/) (2024)
- [Distributional Offline Policy Evaluation with Predictive Error Guarantees](https://mlsanity.com/paper/2302.09456/) (2023)
- [Foundations of Multivariate Distributional Reinforcement Learning](https://mlsanity.com/paper/2409.00328/) (2024)
- [The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation](https://mlsanity.com/paper/2305.18388/) (2023)
- [Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learning](https://mlsanity.com/paper/2309.14246/) (2023)
- [Estimation and Inference in Distributional Reinforcement Learning](https://mlsanity.com/paper/2309.17262/) (2023)
- [Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2402.07598/) (2024)
- [Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning](https://mlsanity.com/paper/2410.11022/) (2024)
- [One-Step Distributional Reinforcement Learning](https://mlsanity.com/paper/2304.14421/) (2023)
- [Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis](https://mlsanity.com/paper/2410.24128/) (2024)
- [A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximation](https://mlsanity.com/paper/2502.14172/) (2025)
- [Value-Distributional Model-Based Reinforcement Learning](https://mlsanity.com/paper/2308.06590/) (2023)
- [A Differential Perspective on Distributional Reinforcement Learning](https://mlsanity.com/paper/2506.03333/) (2025)
- [Distributional Bellman Operators over Mean Embeddings](https://mlsanity.com/paper/2312.07358/) (2023)
- [Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation](https://mlsanity.com/paper/2402.18159/) (2024)
- [Statistical Efficiency and Inference of Quantile Distributional Reinforcement Learning](https://mlsanity.com/paper/2607.08444/) (2026)
- [Variance Control for Distributional Reinforcement Learning](https://mlsanity.com/paper/2307.16152/) (2023)
- [The Kernel Density Integral Transformation](https://mlsanity.com/paper/2309.10194/) (2023)
- [Survey and Tutorial of Reinforcement Learning Methods in Process Systems Engineering](https://mlsanity.com/paper/2510.24272/) (2025)
- [Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation](https://mlsanity.com/paper/2510.26026/) (2025)
- [Policy Evaluation in Distributional LQR](https://mlsanity.com/paper/2303.13657/) (2023)
- [Distributional Reinforcement Learning with Dual Expectile-Quantile Regression](https://mlsanity.com/paper/2305.16877/) (2023)
- [ADDQ: Adaptive Distributional Double Q-Learning](https://mlsanity.com/paper/2506.19478/) (2025)
- [Pure Exploration via Frank-Wolfe Self-Play](https://mlsanity.com/paper/2509.19901/) (2025)
- [Accelerated Distributional Temporal Difference Learning with Linear Function Approximation](https://mlsanity.com/paper/2511.12688/) (2025)
- [Online Inference for Quantile Temporal Difference Learning in Distributional Reinforcement Learning](https://mlsanity.com/paper/2608.12973/) (2026)
- [A Case for Validation Buffer in Pessimistic Actor-Critic](https://mlsanity.com/paper/2403.01014/) (2024)
- [Stochastic Approximation Methods for Distortion Risk Measure Optimization](https://mlsanity.com/paper/2510.04563/) (2025)
- [Distributional Off-Policy Evaluation with Deep Quantile Process Regression](https://mlsanity.com/paper/2604.18143/) (2026)

## Abstract (arXiv)

We analyse quantile temporal-difference learning (QTD), a distributional reinforcement learning algorithm that has proven to be a key component in several successful large-scale applications of reinforcement learning. Despite these empirical successes, a theoretical understanding of QTD has proven elusive until now. Unlike classical TD learning, which can be analysed with standard stochastic approximation tools, QTD updates do not approximate contraction mappings, are highly non-linear, and may have multiple fixed points. The core result of this paper is a proof of convergence to the fixed points of a related family of dynamic programming procedures with probability 1, putting QTD on firm theoretical footing. The proof establishes connections between QTD and non-linear differential inclusions through stochastic approximation theory and non-smooth analysis.