mlsanityData to 14 Oct 2026

Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning

arXiv 2506.15544 ↗First posted 18 Jun 2025cs.LGcited by 23

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Scaling deep reinforcement learning networks is challenging and often results in degraded performance, yet the root causes of this failure mode remain poorly understood. Several recent works have proposed mechanisms to address this, but they are often complex and fail to highlight the causes underlying this difficulty. In this work, we conduct a series of empirical analyses which suggest that the combination of non-stationarity with gradient pathologies, due to suboptimal architectural choices, underlie the challenges of scale. We propose a series of direct interventions that stabilize gradient flow, enabling robust performance across a range of network depths and widths. Our interventions are simple to implement and compatible with well-established algorithms, and result in an effective mechanism that enables strong performance even at large scales. We validate our findings on a variety of agents and suites of environments.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 67 of its 91 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

67 of 91 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning16 Jun
  2. 2025Horizon Reduction Makes RL Scalable4 Jun
  3. 2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks3 Jun
  4. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning29 May
  5. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities19 Mar
  6. 2024Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps22 Dec
  7. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning13 Oct
  8. 2024Neuroplastic Expansion in Deep Reinforcement Learning10 Oct
  9. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL2 Oct
  10. 2024Simplifying Deep Temporal Difference Learning5 Jul
  11. 2024Normalization and effective learning rates in reinforcement learning1 Jul
  12. 2024Weight Clipping for Deep Continual and Reinforcement Learning1 Jul
8 more
  1. 2024Mixture of Experts in a Mixture of RL settings26 Jun
  2. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learning25 Jun
  3. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning29 May
  4. 2024Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning26 May
  5. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May
  6. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO1 May
  7. 2024In value-based deep reinforcement learning, a pruned network is a good network19 Feb
  8. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL13 Feb

Other references · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Densely Connected Convolutional Networkscited by 6,720
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2016Layer Normalizationcited by 5,669
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  10. 2015Continuous control with deep reinforcement learningcited by 3,407
  11. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  12. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
35 more
  1. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  2. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  3. 2018DeepMind Control Suitecited by 956
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  8. 2014New insights and perspectives on the natural gradient methodcited by 443
  9. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  10. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  11. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  12. 2016Deep Information Propagationcited by 195
  13. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  14. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  15. 2023Understanding plasticity in neural networkscited by 174
  16. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  17. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  18. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  19. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  20. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 121
  21. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  22. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 102
  23. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  24. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  25. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 73
  26. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
  27. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  28. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  29. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 42
  30. 2021Training Larger Networks for Deep Reinforcement Learningcited by 38
  31. 2023Small batch deep reinforcement learningcited by 36
  32. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  33. 2021Off-Policy Reinforcement Learning with Delayed Rewardscited by 25
  34. 2020Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimizationcited by 16
  35. 2021Correcting Momentum in Temporal Difference Learningcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 18.

23 papers.

Sort:

By area:

  1. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
  2. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  3. 2026TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapsecited by 9
  4. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  5. 2025Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoningcited by 5
  6. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  7. 2026Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentialscited by 3
  8. 2025Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokkingcited by 2
  9. 2026Squint: Fast Visual Reinforcement Learning for Sim-to-Real Roboticscited by 2
  10. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
  11. 2026Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learningcited by 1
  12. 2026Joint Learning of Hierarchical Neural Options and Abstract World Modelcited by 1
11 more
  1. 2026AdamO: A Collapse-Suppressed Optimizer for Offline RLcited by 1
  2. 2026Agentick: A Unified Benchmark for General Sequential Decision-Making Agentscited by 1
  3. 2026Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Losscited by 1
  4. 2026Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learningcited by 1
  5. 2026Vision Transformers that Never Stop Learningcited by 0
  6. 2026Scalable Reinforcement Learning via Adaptive Batch Scalingcited by 0
  7. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0
  8. 2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learningcited by 0
  9. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0
  10. 2026Understanding and Stabilizing Deep Q-Learning via Controlled Bootstrapping and Regulated Value Dynamicscited by 0
  11. 2026Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Criticcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning
  2. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  3. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Model
  4. 2026Training instability in deep learning follows low-dimensional dynamical principles
  5. 2026GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control
  6. 2025The Art of Scaling Reinforcement Learning Compute for LLMs
  7. 2024The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations
  8. 2025Multi-Task Reinforcement Learning Enables Parameter Scaling