mlsanityData to 14 Oct 2026

Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning

arXiv 2505.24061 ↗First posted 29 May 2025cs.LGcited by 14

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Deep reinforcement learning (RL) agents frequently suffer from neuronal activity loss, which impairs their ability to adapt to new data and learn continually. A common method to quantify and address this issue is the tau-dormant neuron ratio, which uses activation statistics to measure the expressive ability of neurons. While effective for simple MLP-based agents, this approach loses statistical power in more complex architectures. To address this, we argue that in advanced RL agents, maintaining a neuron's learning capacity, its ability to adapt via gradient updates, is more critical than preserving its expressive ability. Based on this insight, we shift the statistical objective from activations to gradients, and introduce GraMa (Gradient Magnitude Neural Activity Metric), a lightweight, architecture-agnostic metric for quantifying neuron-level learning capacity. We show that GraMa effectively reveals persistent neuron inactivity across diverse architectures, including residual networks, diffusion models, and agents with varied activation functions. Moreover, resetting neurons guided by GraMa (ReGraMa) consistently improves learning performance across multiple deep RL algorithms and benchmarks, such as MuJoCo and the DeepMind Control Suite.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 48 of its 68 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

48 of 68 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Multi-Task Reinforcement Learning Enables Parameter Scaling7 Mar
  2. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learning21 Feb
  3. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning13 Oct
  4. 2024Neuroplastic Expansion in Deep Reinforcement Learning10 Oct
  5. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL2 Oct
  6. 2024Mixture of Experts in a Mixture of RL settings26 Jun
  7. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learning25 Jun
  8. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning29 May
  9. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May
  10. 2024Disentangling the Causes of Plasticity Loss in Neural Networks29 Feb
  11. 2024In value-based deep reinforcement learning, a pruned network is a good network19 Feb
  12. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL13 Feb

Other references · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  6. 2018DeepMind Control Suitecited by 956
  7. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  8. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  9. 2024Diffusion Policy Policy Optimizationcited by 280
  10. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  11. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 176
  12. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
24 more
  1. 2023Understanding plasticity in neural networkscited by 174
  2. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  3. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  4. 2024Diffusion Actor-Critic with Entropy Regulatorcited by 94
  5. 2021RRL: Resnet as representation for Reinforcement Learningcited by 91
  6. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  7. 2025Efficient Online Reinforcement Learning for Diffusion Policycited by 71
  8. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 65
  9. 2021Autonomous Drone Racing with Deep Reinforcement Learningcited by 60
  10. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  11. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 50
  12. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 47
  13. 2024Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networkscited by 43
  14. 2018Collapse of Deep and Narrow Neural Netscited by 37
  15. 2021Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Reviewcited by 36
  16. 2023Small batch deep reinforcement learningcited by 36
  17. 2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 34
  18. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  19. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 28
  20. 2025Learning to Sample Effective and Diverse Prompts for Text-to-Image Generationcited by 17
  21. 2024ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularizationcited by 16
  22. 2024Self-Normalized Resets for Plasticity in Continual Learningcited by 11
  23. 2019On Warm-Starting Neural Network Trainingcited by 9
  24. 2024How Does Gradient Descent Learn Features -- A Local Analysis for Regularized Two-Layer Neural Networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 10.

14 papers.

Sort:

By area:

  1. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  2. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  3. 2025The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilitiescited by 7
  4. 2025Activation Function Design Sustains Plasticity in Continual Learningcited by 6
  5. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  6. 2026The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learningcited by 2
  7. 2026Can Scale Save Us From Plasticity Loss in Large Language Models?cited by 1
  8. 2026Attribution-Based Neuron Utility for Plasticity Restoration in Deep Networkscited by 0
  9. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0
  10. 2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learningcited by 0
  11. 2026Stagnant Neuron: Towards Understanding the Plasticity Loss in Multi-Agent Reinforcement Learning Value Factorization Methodscited by 0
  12. 2026Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learningcited by 0
2 more
  1. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0
  2. 2026SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sample-Efficient Neurosymbolic Deep Reinforcement Learning
  2. 2025Neuron-level Balance between Stability and Plasticity in Deep Reinforcement Learning
  3. 2025Mastering Continual Reinforcement Learning through Fine-Grained Sparse Network Allocation and Dormant Neuron Exploration
  4. 2025Silent Neuron Theory and Plasticity Preservation for Deep Reinforcement Learning in Adaptive Video Streaming
  5. 2025Optimizing Sensory Neurons: Nonlinear Attention Mechanisms for Accelerated Convergence in Permutation-Invariant Neural Networks for Reinforcement Learning
  6. 2025Understanding Behavioral Metric Learning: A Large-Scale Study on Distracting Reinforcement Learning Environments
  7. 2026Signal-Adaptive Trust Regions for Gradient-Free Optimization of Recurrent Spiking Neural Networks
  8. 2026Rethinking Plasticity in Deep Reinforcement Learning