mlsanityData to 14 Oct 2026

Faster Deep Reinforcement Learning with Slower Online Network

arXiv 2112.05848 ↗First posted 10 Dec 2021cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning algorithms often use two networks for value function optimization: an online network, and a target network that tracks the online network with some delay. Using two separate networks enables the agent to hedge against issues that arise when performing bootstrapping. In this paper we endow two popular deep reinforcement learning algorithms, namely DQN and Rainbow, with updates that incentivize the online network to remain in the proximity of the target network. This improves the robustness of deep reinforcement learning in presence of noisy updates. The resultant agents, called DQN Pro and Rainbow Pro, exhibit significant performance improvements over their original counterparts on the Atari benchmark demonstrating the effectiveness of this simple idea in deep reinforcement learning. The code for our paper is available here: Github.com/amazon-research/fast-rl-with-slow-updates.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 28 of its 68 references that are in the map. First posted on arXiv 10 Dec 2021.

Builds on

28 of 68 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  7. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  8. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  9. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  10. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  11. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  12. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
16 more
  1. 2020Mirror Descent Policy Optimizationcited by 106
  2. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 91
  3. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 82
  4. 2019Meta-Q-Learningcited by 62
  5. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  6. 2015Incremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Surveycited by 45
  7. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
  8. 2019Trust Region-Guided Proximal Policy Optimizationcited by 36
  9. 2017Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learningcited by 32
  10. 2020Finite-Sample Analysis of Proximal Gradient TD Algorithmscited by 32
  11. 2019Target-Based Temporal Difference Learningcited by 20
  12. 2014Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spacescited by 19
  13. 2015Proximal Algorithms in Statistics and Machine Learningcited by 18
  14. 2019Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learningcited by 16
  15. 2018Natural Gradient Deep Q-learningcited by 5
  16. 2020Proximal Deterministic Policy Gradientcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2; 2025: 2.

7 papers.

  1. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  2. 2023TD Convergence: An Optimization Perspectivecited by 16
  3. 2024Reinforcement Learning From Imperfect Corrective Actions And Proxy Rewardscited by 3
  4. 2025On the Interplay Between Sparsity and Training in Deep Reinforcement Learningcited by 3
  5. 2023Counterfactual Explanation Policies in RLcited by 2
  6. 2024Learning the Target Network in Function Spacecited by 1
  7. 2025Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Driftcited by 1