mlsanityData to 14 Oct 2026

Parseval Regularization for Continual Reinforcement Learning

arXiv 2412.07224 ↗First posted 10 Dec 2024cs.LG cs.AIcited by 21

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Loss of plasticity, trainability loss, and primacy bias have been identified as issues arising when training deep neural networks on sequences of tasks -- all referring to the increased difficulty in training on new tasks. We propose to use Parseval regularization, which maintains orthogonality of weight matrices, to preserve useful optimization properties and improve training in a continual reinforcement learning setting. We show that it provides significant benefits to RL agents on a suite of gridworld, CARL and MetaWorld tasks. We conduct comprehensive ablations to identify the source of its benefits and investigate the effect of certain metrics associated to network trainability including weight matrix rank, weight norms and policy entropy.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 49 of its 80 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

49 of 80 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2016Layer Normalizationcited by 5,669
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2017Improved Training of Wasserstein GANscited by 2,235
  9. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  11. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  12. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
37 more
  1. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  2. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  3. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  4. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  5. 2020What is being transferred in transfer learning?cited by 368
  6. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 293
  7. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  8. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  9. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  10. 2018Sorting out Lipschitz function approximationcited by 213
  11. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  12. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  13. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  14. 2023Understanding plasticity in neural networkscited by 174
  15. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 157
  16. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  17. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  18. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  19. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  20. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  21. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  22. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  23. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  24. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 80
  25. 2017Critical Learning Periods in Deep Neural Networkscited by 79
  26. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  27. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  28. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  29. 2021Skew Orthogonal Convolutionscited by 67
  30. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 62
  31. 2023Continual Learning: Applications and the Road Forwardcited by 59
  32. 2022Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networkscited by 39
  33. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  34. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  35. 2021CARL: A Benchmark for Contextual and Adaptive Reinforcement Learningcited by 21
  36. 2022Robust Policy Optimization in Deep Reinforcement Learningcited by 15
  37. 2020Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization?cited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 12.

21 papers.

Sort:

By area:

  1. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  2. 2025MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learningcited by 11
  3. 2026FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoffcited by 7
  4. 2025Spectral Collapse Drives Loss of Plasticity in Deep Continual Learningcited by 6
  5. 2026Rotation-Preserving Supervised Fine-Tuningcited by 3
  6. 2025Overcoming Non-stationary Dynamics with Evidential Proximal Policy Optimizationcited by 2
  7. 2025Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learningcited by 2
  8. 2025Preserving Plasticity in Continual Learning with Adaptive Linearity Injectioncited by 2
  9. 2025Sparsity-Driven Plasticity in Multi-Task Reinforcement Learningcited by 2
  10. 2026Do Neural Networks Lose Plasticity in a Gradually Changing World?cited by 2
  11. 2026The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learningcited by 2
  12. 2026SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learningcited by 2
9 more
  1. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
  2. 2026Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learningcited by 1
  3. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learningcited by 1
  4. 2026Can Scale Save Us From Plasticity Loss in Large Language Models?cited by 1
  5. 2025ADARL: Adaptive Low-Rank Structures for Robust Policy Learning under Uncertaintycited by 0
  6. 2026Policy Gradient with Adaptive Entropy Annealing for Continual Fine-Tuningcited by 0
  7. 2026CDRL: A Reinforcement Learning Framework Inspired by Cerebellar Circuits and Dendritic Computational Strategiescited by 0
  8. 2026Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learningcited by 0
  9. 2026SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning Continually by Spectral Regularization
  2. 2024Weight Clipping for Deep Continual and Reinforcement Learning
  3. 2026Don't Forget the Critic: Value-Based Data Rehearsal for Multi-Cyclic Continual Reinforcement Learning
  4. 2026A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning
  5. 2023A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning
  6. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization
  7. 2025Convergence of regularized agent-state-based Q-learning in POMDPs
  8. 2024Data Augmentation for Continual RL via Adversarial Gradient Episodic Memory