# Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Target

arXiv: https://arxiv.org/abs/1901.07510 · HTML: https://mlsanity.com/paper/1901.07510/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jan 2019.

## Built on by (most cited first, 30 of 31)

- [Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL](https://mlsanity.com/paper/2209.03993/) (2022)
- [Empirical Design in Reinforcement Learning](https://mlsanity.com/paper/2304.01315/) (2023)
- [Stabilizing Off-Policy Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2207.00986/) (2022)
- [RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch](https://mlsanity.com/paper/2205.15043/) (2022)
- [A Technical Survey of Reinforcement Learning Techniques for Large Language Models](https://mlsanity.com/paper/2507.04136/) (2025)
- [Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2006.16785/) (2020)
- [Averaging n-step Returns Reduces Variance in Reinforcement Learning](https://mlsanity.com/paper/2402.03903/) (2024)
- [Skill or Luck? Return Decomposition via Advantage Functions](https://mlsanity.com/paper/2402.12874/) (2024)
- [Transductive Off-policy Proximal Policy Optimization](https://mlsanity.com/paper/2406.03894/) (2024)
- [Reflective Policy Optimization](https://mlsanity.com/paper/2406.03678/) (2024)
- [Chunk-Guided Q-Learning](https://mlsanity.com/paper/2603.13971/) (2026)
- [Highway Reinforcement Learning](https://mlsanity.com/paper/2405.18289/) (2024)
- [Offline Reinforcement Learning with Universal Horizon Models](https://mlsanity.com/paper/2605.15603/) (2026)
- [Policies for the Dynamic Traveling Maintainer Problem with Alerts](https://mlsanity.com/paper/2105.15119/) (2021)
- [Graph Backup: Data Efficient Backup Exploiting Markovian Transitions](https://mlsanity.com/paper/2205.15824/) (2022)
- [Efficient Open-world Reinforcement Learning via Knowledge Distillation and Autonomous Rule Discovery](https://mlsanity.com/paper/2311.14270/) (2023)
- [Analysis of Off-Policy Multi-Step TD-Learning with Linear Function Approximation](https://mlsanity.com/paper/2402.15781/) (2024)
- [Beyond Monotonicity: Revisiting Factorization Principles in Multi-Agent Q-Learning](https://mlsanity.com/paper/2511.09792/) (2025)
- [Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2606.20411/) (2026)
- [Data-Driven Evaluation of Training Action Space for Reinforcement Learning](https://mlsanity.com/paper/2204.03840/) (2022)
- [Long N-step Surrogate Stage Reward to Reduce Variances of Deep Reinforcement Learning in Complex Problems](https://mlsanity.com/paper/2210.04820/) (2022)
- [Mining of Single-Class by Active Learning for Semantic Segmentation](https://mlsanity.com/paper/2307.09109/) (2023)
- [An Improved Strategy for Blood Glucose Control Using Multi-Step Deep Reinforcement Learning](https://mlsanity.com/paper/2403.07566/) (2024)
- [Monocular Obstacle Avoidance Based on Inverse PPO for Fixed-wing UAVs](https://mlsanity.com/paper/2411.18009/) (2024)
- [Analysis of Off-Policy n-Step TD-Learning with Linear Function Approximation](https://mlsanity.com/paper/2502.08941/) (2025)
- [Ensemble Elastic DQN: A Step Dependent Ensemble Approach for Reducing Overestimation in Deep Value-Based Reinforcement Learning](https://mlsanity.com/paper/2506.05716/) (2025)
- [The Role of Target Update Frequencies in Q-Learning](https://mlsanity.com/paper/2602.03911/) (2026)
- [Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities](https://mlsanity.com/paper/2605.05812/) (2026)
- [Memory Merge DQN: Sensitivity Weighted Target Updates for Stable Value Learning](https://mlsanity.com/paper/2607.19397/) (2026)
- [Gated Q-learning: Add Off-Policy Bias to Taste](https://mlsanity.com/paper/2607.28916/) (2026)

## Abstract (arXiv)

Multi-step methods such as Retrace(λ) and n-step Q-learning have become a crucial component of modern deep reinforcement learning agents. These methods are often evaluated as a part of bigger architectures and their evaluations rarely include enough samples to draw statistically significant conclusions about their performance. This type of methodology makes it difficult to understand how particular algorithmic details of multi-step methods influence learning. In this paper we combine the n-step action-value algorithms Retrace, Q-learning, Tree Backup, Sarsa, and Q(σ) with an architecture analogous to DQN. We test the performance of all these algorithms in the mountain car environment; this choice of environment allows for faster training times and larger sample sizes. We present statistical analyses on the effects of the off-policy correction, the backup length parameter n, and the update frequency of the target network on the performance of these algorithms. Our results show that (1) using off-policy correction can have an adverse effect on the performance of Sarsa and Q(σ); (2) increasing the backup length n consistently improved performance across all the different algorithms; and (3) the performance of Sarsa and Q-learning was more robust to the effect of the target network update frequency than the performance of Tree Backup, Q(σ), and Retrace in this particular task.