# Demystifying the Recency Heuristic in Temporal-Difference Learning

arXiv: https://arxiv.org/abs/2406.12284 · HTML: https://mlsanity.com/paper/2406.12284/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 12 of its 31 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 12 of 12)

- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Safe and Efficient Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1606.02647/) (2016)
- [Q(λ) with Off-Policy Corrections](https://mlsanity.com/paper/1602.04951/) (2016)
- [A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning](https://mlsanity.com/paper/1607.00446/) (2016)
- [Multi-step Off-policy Learning Without Importance Sampling Ratios](https://mlsanity.com/paper/1702.03006/) (2017)
- [Revisiting Peng's Q(λ) for Modern Reinforcement Learning](https://mlsanity.com/paper/2103.00107/) (2021)
- [Averaging n-step Returns Reduces Variance in Reinforcement Learning](https://mlsanity.com/paper/2402.03903/) (2024)
- [Investigating Recurrence and Eligibility Traces in Deep Q-Networks](https://mlsanity.com/paper/1704.05495/) (2017)
- [Effective Multi-step Temporal-Difference Learning for Non-Linear Function Approximation](https://mlsanity.com/paper/1608.05151/) (2016)
- [Trajectory-Aware Eligibility Traces for Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2301.11321/) (2023)
- [From Past to Future: Rethinking Eligibility Traces](https://mlsanity.com/paper/2312.12972/) (2023)
- [Off-policy Distributional Q(λ): Distributional RL without Importance Sampling](https://mlsanity.com/paper/2402.05766/) (2024)

## Built on by (most cited first, 4 of 4)

- [Deep Reinforcement Learning with Gradient Eligibility Traces](https://mlsanity.com/paper/2507.09087/) (2025)
- [Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning](https://mlsanity.com/paper/2602.12375/) (2026)
- [To bootstrap or to rollout? An optimal and adaptive interpolation](https://mlsanity.com/paper/2411.09731/) (2024)
- [Reward Inflation: A Healthy Stimulus for Reinforcement Learning](https://mlsanity.com/paper/2610.02545/) (2026)

## Abstract (arXiv)

The recency heuristic in reinforcement learning is the assumption that stimuli that occurred closer in time to an acquired reward should be more heavily reinforced. The recency heuristic is one of the key assumptions made by TD(λ), which reinforces recent experiences according to an exponentially decaying weighting. In fact, all other widely used return estimators for TD learning, such as n-step returns, satisfy a weaker (i.e., non-monotonic) recency heuristic. Why is the recency heuristic effective for temporal credit assignment? What happens when credit is assigned in a way that violates this heuristic? In this paper, we analyze the specific mathematical implications of adopting the recency heuristic in TD learning. We prove that any return estimator satisfying this heuristic: 1) is guaranteed to converge to the correct value function, 2) has a relatively fast contraction rate, and 3) has a long window of effective credit assignment, yet bounded worst-case variance. We also give a counterexample where on-policy, tabular TD methods violating the recency heuristic diverge. Our results offer some of the first theoretical evidence that credit assignment based on the recency heuristic facilitates learning.