mlsanityData to 7 Oct 2026

The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learning

arXiv 2207.07570 ↗First posted 15 Jul 2022cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We study the multi-step off-policy learning approach to distributional RL. Despite the apparent similarity between value-based RL and distributional RL, our study reveals intriguing and fundamental differences between the two cases in the multi-step setting. We identify a novel notion of path-dependent distributional TD error, which is indispensable for principled multi-step distributional RL. The distinction from the value-based case bears important implications on concepts such as backward-view algorithms. Our work provides the first theoretical guarantees on multi-step off-policy distributional RL algorithms, including results that apply to the small number of existing approaches to multi-step distributional RL. In addition, we derive a novel algorithm, Quantile Regression-Retrace, which leads to a deep RL agent QR-DQN-Retrace that shows empirical improvements over QR-DQN on the Atari-57 benchmark. Collectively, we shed light on how unique challenges in multi-step distributional RL can be addressed both in theory and practice.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 19 of its 40 references that are in the map. First posted on arXiv 15 Jul 2022.

Builds on

19 of 40 references.

References · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  3. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  4. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  6. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  7. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  8. 2017The Cramer Distance as a Solution to Biased Wasserstein Gradientscited by 116
  9. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
  10. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  11. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  12. 2016Q(λ) with Off-Policy Correctionscited by 48
7 more
  1. 2021Universal Off-Policy Evaluationcited by 43
  2. 2019Quantile QT-Opt for Risk-Aware Vision-Based Robotic Graspingcited by 27
  3. 2020Expected Eligibility Tracescited by 27
  4. 2017Multi-step Off-policy Learning Without Importance Sampling Ratioscited by 19
  5. 2019Adaptive Trade-Offs in Off-Policy Learningcited by 19
  6. 2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
  7. 2021GMAC: A Distributional Perspective on Actor-Critic Frameworkcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 1; 2026: 2.

9 papers.

  1. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  2. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
  3. 2023Constrained Reinforcement Learning using Distributional Representation for Trustworthy Quadrotor UAV Tracking Controlcited by 5
  4. 2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 5
  5. 2024Reflective Policy Optimizationcited by 2
  6. 2024Off-policy Distributional Q(λ): Distributional RL without Importance Samplingcited by 1
  7. 2023Bias Resilient Multi-Step Off-Policy Goal-Conditioned Reinforcement Learningcited by 0
  8. 2026Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learningcited by 0
  9. 2026Revisiting TD Target Aggregation under Uncertainty in Q-Learningcited by 0