# Accelerated Distributional Temporal Difference Learning with Linear Function Approximation

arXiv: https://arxiv.org/abs/2511.12688 · HTML: https://mlsanity.com/paper/2511.12688/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 16 Nov 2025.

## Builds on (most cited first, 28 of 28)

- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximation](https://mlsanity.com/paper/1806.02450/) (2018)
- [Minimizing Finite Sums with the Stochastic Average Gradient](https://mlsanity.com/paper/1309.2388/) (2013)
- [Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysis](https://mlsanity.com/paper/2102.06548/) (2021)
- [An Analysis of Categorical Distributional Reinforcement Learning](https://mlsanity.com/paper/1802.08163/) (2018)
- [A Comparative Analysis of Expected and Distributional Reinforcement Learning](https://mlsanity.com/paper/1901.11084/) (2019)
- [Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis](https://mlsanity.com/paper/2003.07337/) (2020)
- [Simple and optimal methods for stochastic variational inequalities, I: operator extrapolation](https://mlsanity.com/paper/2011.02987/) (2020)
- [Optimal and instance-dependent guarantees for Markovian linear stochastic approximation](https://mlsanity.com/paper/2112.12770/) (2021)
- [An Analysis of Quantile Temporal-Difference Learning](https://mlsanity.com/paper/2301.04462/) (2023)
- [Finite time analysis of temporal difference learning with linear function approximation: Tail averaging and regularisation](https://mlsanity.com/paper/2210.05918/) (2022)
- [Distributional Offline Policy Evaluation with Predictive Error Guarantees](https://mlsanity.com/paper/2302.09456/) (2023)
- [Improved High-Probability Bounds for the Temporal Difference Learning Algorithm via Exponential Stability](https://mlsanity.com/paper/2310.14286/) (2023)
- [Gaussian Approximation and Multiplier Bootstrap for Polyak-Ruppert Averaged Linear Stochastic Approximation with Applications to TD Learning](https://mlsanity.com/paper/2405.16644/) (2024)
- [Accelerated and instance-optimal policy evaluation with linear function approximation](https://mlsanity.com/paper/2112.13109/) (2021)
- [Foundations of Multivariate Distributional Reinforcement Learning](https://mlsanity.com/paper/2409.00328/) (2024)
- [Distributional reinforcement learning with linear function approximation](https://mlsanity.com/paper/1902.03149/) (2019)
- [High-probability sample complexities for policy evaluation with linear function approximation](https://mlsanity.com/paper/2305.19001/) (2023)
- [Statistical Inference for Policy Evaluation with Temporal Difference Learning](https://mlsanity.com/paper/2410.16106/) (2024)
- [Optimizing Return Distributions with Distributional Dynamic Programming](https://mlsanity.com/paper/2501.13028/) (2025)
- [Distributional Model Equivalence for Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2307.01708/) (2023)
- [Estimation and Inference in Distributional Reinforcement Learning](https://mlsanity.com/paper/2309.17262/) (2023)
- [Statistical Efficiency of Distributional Temporal Difference Learning and Freedman's Inequality in Hilbert Spaces](https://mlsanity.com/paper/2403.05811/) (2024)
- [Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2402.07598/) (2024)
- [Nonlinear Distributional Gradient Temporal-Difference Learning](https://mlsanity.com/paper/1805.07732/) (2018)
- [Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning](https://mlsanity.com/paper/2501.02087/) (2025)
- [A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximation](https://mlsanity.com/paper/2502.14172/) (2025)

## Built on by (most cited first, 1 of 1)

- [Online Inference in Distributional Temporal-Difference Learning](https://mlsanity.com/paper/2608.14408/) (2026)

## Abstract (arXiv)

In this paper, we study the finite-sample statistical rates of distributional temporal difference (TD) learning with linear function approximation. The purpose of distributional TD learning is to estimate the return distribution of a discounted Markov decision process for a given policy. Previous works on statistical analysis of distributional TD learning focus mainly on the tabular case. We first consider the linear function approximation setting and conduct a fine-grained analysis of the linear-categorical Bellman equation. Building on this analysis, we further incorporate variance reduction techniques in our new algorithms to establish tight sample complexity bounds independent of the support size K when K is large. Our theoretical results imply that, when employing distributional TD learning with linear function approximation, learning the full distribution of the return function from streaming data is no more difficult than learning its expectation. This work provide new insights into the statistical efficiency of distributional reinforcement learning algorithms.