mlsanityData to 14 Oct 2026

Demystifying the Recency Heuristic in Temporal-Difference Learning

arXiv 2406.12284 ↗First posted 18 Jun 2024cs.LG cs.AIcited by 4

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

The recency heuristic in reinforcement learning is the assumption that stimuli that occurred closer in time to an acquired reward should be more heavily reinforced. The recency heuristic is one of the key assumptions made by TD(λ), which reinforces recent experiences according to an exponentially decaying weighting. In fact, all other widely used return estimators for TD learning, such as n-step returns, satisfy a weaker (i.e., non-monotonic) recency heuristic. Why is the recency heuristic effective for temporal credit assignment? What happens when credit is assigned in a way that violates this heuristic? In this paper, we analyze the specific mathematical implications of adopting the recency heuristic in TD learning. We prove that any return estimator satisfying this heuristic: 1) is guaranteed to converge to the correct value function, 2) has a relatively fast contraction rate, and 3) has a long window of effective credit assignment, yet bounded worst-case variance. We also give a counterexample where on-policy, tabular TD methods violating the recency heuristic diverge. Our results offer some of the first theoretical evidence that credit assignment based on the recency heuristic facilitates learning.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 31 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

12 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Averaging n-step Returns Reduces Variance in Reinforcement Learning6 Feb

Other references · 11

  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  2. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  3. 2016Q(λ) with Off-Policy Correctionscited by 48
  4. 2016A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learningcited by 20
  5. 2017Multi-step Off-policy Learning Without Importance Sampling Ratioscited by 19
  6. 2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
  7. 2017Investigating Recurrence and Eligibility Traces in Deep Q-Networkscited by 8
  8. 2016Effective Multi-step Temporal-Difference Learning for Non-Linear Function Approximationcited by 7
  9. 2023Trajectory-Aware Eligibility Traces for Off-Policy Reinforcement Learningcited by 6
  10. 2023From Past to Future: Rethinking Eligibility Tracescited by 3
  11. 2024Off-policy Distributional Q(λ): Distributional RL without Importance Samplingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2025Deep Reinforcement Learning with Gradient Eligibility Tracescited by 6
  2. 2026Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learningcited by 1
  3. 2024To bootstrap or to rollout? An optimal and adaptive interpolationcited by 0
  4. 2026Reward Inflation: A Healthy Stimulus for Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning
  2. 2025Stabilizing Temporal Difference Learning via Implicit Stochastic Recursion
  3. 2026Towards Parameter-Free Temporal Difference Learning
  4. 2025Adversarially-Robust TD Learning with Markovian Data: Finite-Time Rates and Fundamental Limits
  5. 2025Implicit Updates for Average-Reward Temporal Difference Learning
  6. 2026Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes
  7. 2024Statistical Inference for Policy Evaluation with Temporal Difference Learning
  8. 2026Counterfactual Shapley Credit Assignment