mlsanityData to 14 Oct 2026

Resetting the Optimizer in Deep RL: An Empirical Study

arXiv 2306.17833 ↗First posted 30 Jun 2023cs.LG cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

We focus on the task of approximating the optimal value function in deep reinforcement learning. This iterative process is comprised of solving a sequence of optimization problems where the loss function changes per iteration. The common approach to solving this sequence of problems is to employ modern variants of the stochastic gradient descent algorithm such as Adam. These optimizers maintain their own internal parameters such as estimates of the first-order and the second-order moments of the gradient, and update them over time. Therefore, information obtained in previous iterations is used to solve the optimization problem in the current iteration. We demonstrate that this can contaminate the moment estimates because the optimization landscape can change arbitrarily from one iteration to the next one. To hedge against this negative effect, a simple idea is to reset the internal parameters of the optimizer when starting a new iteration. We empirically investigate this resetting idea by employing various optimizers in conjunction with the Rainbow algorithm. We demonstrate that this simple modification significantly improves the performance of deep RL on the Atari benchmark.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 28 of its 53 references that are in the map. First posted on arXiv 30 Jun 2023.

Builds on

28 of 53 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2017Decoupled Weight Decay Regularizationcited by 1,831
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  6. 2015Prioritized Experience Replaycited by 1,099
  7. 2016An overview of gradient descent optimization algorithmscited by 1,033
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  9. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  10. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  11. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  12. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
16 more
  1. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  2. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  3. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  4. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  5. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  6. 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
  7. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  8. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  9. 2017Multi-step Reinforcement Learning: A Unifying Algorithmcited by 42
  10. 2020Scheduled Restart Momentum for Accelerated Stochastic Gradient Descentcited by 24
  11. 2020Lipschitz Lifelong Reinforcement Learningcited by 23
  12. 2021Correcting Momentum in Temporal Difference Learningcited by 12
  13. 2022Meta-Gradients in Non-Stationary Environmentscited by 11
  14. 2022The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learningcited by 10
  15. 2021Faster Deep Reinforcement Learning with Slower Online Networkcited by 7
  16. 2020Restarting Algorithms: Sometimes there is Free Lunchcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 5; 2026: 10.

26 papers.

Sort:

By area:

  1. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 86
  2. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPOcited by 27
  3. 2024Streaming Deep Reinforcement Learning Finally Workscited by 25
  4. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  5. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  6. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  7. 2024Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learningcited by 17
  8. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  9. 2024Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timestepscited by 11
  10. 2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learningcited by 11
  11. 2025Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Losscited by 6
  12. 2025Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learningcited by 6
14 more
  1. 2026Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resetscited by 6
  2. 2024Reinforcement Learning From Imperfect Corrective Actions And Proxy Rewardscited by 3
  3. 2024Beyond the Boundaries of Proximal Policy Optimizationcited by 3
  4. 2026Do Neural Networks Lose Plasticity in a Gradually Changing World?cited by 2
  5. 2026The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learningcited by 2
  6. 2024Learning the Target Network in Function Spacecited by 1
  7. 2024Uncertainty Quantification with Bayesian Higher Order ReLU KANscited by 1
  8. 2026Rethinking Adam for Time Series Forecasting: A Simple Heuristic to Improve Optimization under Distribution Shiftscited by 1
  9. 2026AdamO: A Collapse-Suppressed Optimizer for Offline RLcited by 1
  10. 2026Can Scale Save Us From Plasticity Loss in Large Language Models?cited by 1
  11. 2026The Role of Target Update Frequencies in Q-Learningcited by 0
  12. 2026Rethinking Plasticity in Deep Reinforcement Learningcited by 0
  13. 2026On the Stability of Growth in Structural Plasticity: Forward-Active yet Backward-Starvedcited by 0
  14. 2026SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learningcited by 0