mlsanityData to 14 Oct 2026

Dissecting Deep RL with High Update Ratios: Combatting Value Divergence

arXiv 2403.05996 ↗First posted 9 Mar 2024cs.LG cs.AIcited by 17

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

We show that deep reinforcement learning algorithms can retain their ability to learn without resetting network parameters in settings where the number of gradient updates greatly exceeds the number of environment samples by combatting value function divergence. Under large update-to-data ratios, a recent study by Nikishin et al. (2022) suggested the emergence of a primacy bias, in which agents overfit early interactions and downplay later experience, impairing their ability to learn. In this work, we investigate the phenomena leading to the primacy bias. We inspect the early stages of training that were conjectured to cause the failure to learn and find that one fundamental challenge is a long-standing acquaintance: value function divergence. Overinflated Q-values are found not only on out-of-distribution but also in-distribution data and can be linked to overestimation on unseen action prediction propelled by optimizer momentum. We employ a simple unit-ball normalization that enables learning under large update ratios, show its efficacy on the widely used dm_control suite, and obtain strong performance on the challenging dog tasks, competitive with model-based approaches. Our results question, in parts, the prior explanation for sub-optimal learning due to overfitting early data.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 49 of its 77 references that are in the map. First posted on arXiv 9 Mar 2024.

Builds on

49 of 77 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Disentangling the Causes of Plasticity Loss in Neural Networks29 Feb

Other references · 48

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  6. 2019Root Mean Square Layer Normalizationcited by 1,480
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  11. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  12. 2017A Distributional Perspective on Reinforcement Learningcited by 765
36 more
  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  2. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  3. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  4. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  5. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  6. 2019Understanding and Improving Layer Normalizationcited by 217
  7. 2020Revisiting Fundamentals of Experience Replaycited by 175
  8. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  9. 2023Understanding plasticity in neural networkscited by 174
  10. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
  11. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
  12. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  13. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  14. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  15. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  16. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  17. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  18. 2018Generalization and Regularization in DQNcited by 118
  19. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  20. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  21. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  22. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  23. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  24. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  25. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  26. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  27. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  28. 2021Ensemble Bootstrapping for Q-Learningcited by 31
  29. 2023Bridging RL Theory and Practice with the Effective Horizoncited by 31
  30. 2021Is High Variance Unavoidable in RL? A Case Study in Continuous Controlcited by 28
  31. 2022The Phenomenon of Policy Churncited by 28
  32. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  33. 2019Harnessing Structures for Value-Based Planning and Reinforcement Learningcited by 24
  34. 2019Momentum in Reinforcement Learningcited by 19
  35. 2023Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agentscited by 12
  36. 2021Estimation Error Correction in Deep Reinforcement Learning for Deterministic Actor-Critic Methodscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 7.

17 papers.

Sort:

By area:

  1. 2024Normalization and effective learning rates in reinforcement learningcited by 58
  2. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 49
  3. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  4. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
  5. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  6. 2025Relative Entropy Pathwise Policy Optimizationcited by 7
  7. 2025Average-Reward Soft Actor-Criticcited by 6
  8. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  9. 2024Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learningcited by 4
  10. 2026What Does Flow Matching Bring To TD Learning?cited by 2
  11. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 1
  12. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theorycited by 1
5 more
  1. 2025Symmetry-Aware Steering of Equivariant Diffusion Policies: Benefits and Limitscited by 1
  2. 2026Behavior-Consistent Deep Reinforcement Learningcited by 1
  3. 2026Distributional Value Estimation Without Target Networks for Robust Quality-Diversitycited by 0
  4. 2026SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learningcited by 0
  5. 2026Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Criticcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL
  2. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
  3. 2023Mind the Model, Not the Agent: The Primacy Bias in Model-based RL
  4. 2025Compute-Optimal Scaling for Value-Based Deep RL
  5. 2023Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting
  6. 2025Value-Based Deep RL Scales Predictably
  7. 2026TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse
  8. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization