mlsanityData to 14 Oct 2026

Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages

arXiv 2310.07418 ↗First posted 11 Oct 2023cs.LG cs.AIcited by 28

Not on the map. Why?

Cited by 20 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Plasticity, the ability of a neural network to evolve with new data, is crucial for high-performance and sample-efficient visual reinforcement learning (VRL). Although methods like resetting and regularization can potentially mitigate plasticity loss, the influences of various components within the VRL framework on the agent's plasticity are still poorly understood. In this work, we conduct a systematic empirical exploration focusing on three primary underexplored facets and derive the following insightful conclusions: (1) data augmentation is essential in maintaining plasticity; (2) the critic's plasticity loss serves as the principal bottleneck impeding efficient training; and (3) without timely intervention to recover critic's plasticity in the early stages, its loss becomes catastrophic. These insights suggest a novel strategy to address the high replay ratio (RR) dilemma, where exacerbated plasticity loss hinders the potential improvements of sample efficiency brought by increased reuse frequency. Rather than setting a static RR for the entire training process, we propose Adaptive RR, which dynamically adjusts the RR based on the critic's plasticity level. Extensive evaluations indicate that Adaptive RR not only avoids catastrophic plasticity loss in the early stages but also benefits from more frequent reuse in later phases, resulting in superior sample efficiency.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 54 of its 62 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

54 of 62 references.

References · 54

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2016Layer Normalizationcited by 5,669
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  5. 2018DeepMind Control Suitecited by 956
  6. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  7. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  8. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  9. 2019Model-Based Reinforcement Learning for Ataricited by 502
  10. 2020Reinforcement Learning with Augmented Datacited by 468
  11. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  12. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
38 more
  1. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  2. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  3. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  4. 2022Masked Visual Pre-training for Motor Controlcited by 248
  5. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  6. 2020Revisiting Fundamentals of Experience Replaycited by 175
  7. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  8. 2023Understanding plasticity in neural networkscited by 174
  9. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  10. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  11. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  12. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  13. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  14. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  15. 2021RRL: Resnet as representation for Reinforcement Learningcited by 91
  16. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  17. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 83
  18. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 80
  19. 2020Deep Reinforcement and InfoMax Learningcited by 79
  20. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  21. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 60
  22. 2020Predictive Information Accelerates Learning in RLcited by 56
  23. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 56
  24. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 52
  25. 2023Maintaining Plasticity in Deep Continual Learningcited by 43
  26. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  27. 2021PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learningcited by 41
  28. 2022Mask-based Latent Reconstruction for Reinforcement Learningcited by 39
  29. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  30. 2020Masked Contrastive Representation Learning for Reinforcement Learningcited by 32
  31. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  32. 2021DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneckcited by 31
  33. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 25
  34. 2021Learning Representations for Pixel-based Control: What Matters and Why?cited by 24
  35. 2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
  36. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
  37. 2021Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Modelcited by 16
  38. 2023New Insights for the Stability-Plasticity Dilemma in Online Continual Learningcited by 16

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 12; 2026: 8.

28 papers.

Sort:

By area:

  1. 2024Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biasescited by 30
  2. 2024Streaming Deep Reinforcement Learning Finally Workscited by 25
  3. 2024UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AIcited by 25
  4. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  5. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  6. 2024Neuroplastic Expansion in Deep Reinforcement Learningcited by 18
  7. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  8. 2024Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrappingcited by 14
  9. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  10. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
  11. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  12. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11
16 more
  1. 2025A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Controlcited by 9
  2. 2026FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoffcited by 7
  3. 2025Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Losscited by 6
  4. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
  5. 2025Optimistic critics can empower small actorscited by 4
  6. 2026Predicting Plasticity in Deep Continual Learning: A Theoretical Perspectivecited by 4
  7. 2025Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learningcited by 2
  8. 2026Do Neural Networks Lose Plasticity in a Gradually Changing World?cited by 2
  9. 2026SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learningcited by 2
  10. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
  11. 2026Angel or Demon: Investigating the Plasticity Interventions' Impact on Backdoor Threats in Deep Reinforcement Learningcited by 1
  12. 2025Neuron-level Balance between Stability and Plasticity in Deep Reinforcement Learningcited by 0
  13. 2025A Visual Reinforcement Learning-Based Separate Primitive Policy for Peg-in-Hole Taskscited by 0
  14. 2026Loss Smoothing for Stable Adaptation Under Distribution Shiftcited by 0
  15. 2026Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Controlcited by 0
  16. 2026Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptationcited by 0