# Investigating Recurrence and Eligibility Traces in Deep Q-Networks

arXiv: https://arxiv.org/abs/1704.05495 · HTML: https://mlsanity.com/paper/1704.05495/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Apr 2017.

## Built on by (most cited first, 8 of 8)

- [Streaming Deep Reinforcement Learning Finally Works](https://mlsanity.com/paper/2410.14606/) (2024)
- [Average-reward model-free reinforcement learning: a systematic review and literature mapping](https://mlsanity.com/paper/2010.08920/) (2020)
- [Revisiting Peng's Q(λ) for Modern Reinforcement Learning](https://mlsanity.com/paper/2103.00107/) (2021)
- [Averaging n-step Returns Reduces Variance in Reinforcement Learning](https://mlsanity.com/paper/2402.03903/) (2024)
- [Demystifying the Recency Heuristic in Temporal-Difference Learning](https://mlsanity.com/paper/2406.12284/) (2024)
- [Two-Step Q-Learning](https://mlsanity.com/paper/2407.02369/) (2024)
- [Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning](https://mlsanity.com/paper/2605.14779/) (2026)
- [Gated Q-learning: Add Off-Policy Bias to Taste](https://mlsanity.com/paper/2607.28916/) (2026)

## Abstract (arXiv)

Eligibility traces in reinforcement learning are used as a bias-variance trade-off and can often speed up training time by propagating knowledge back over time-steps in a single update. We investigate the use of eligibility traces in combination with recurrent networks in the Atari domain. We illustrate the benefits of both recurrent nets and eligibility traces in some Atari games, and highlight also the importance of the optimization used in the training.