# Parseval Regularization for Continual Reinforcement Learning

arXiv: https://arxiv.org/abs/2412.07224 · HTML: https://mlsanity.com/paper/2412.07224/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 49 of its 80 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [ReZero is All You Need: Fast Convergence at Large Depth](https://mlsanity.com/paper/2003.04887/) (2020)
- [The Primacy Bias in Deep Reinforcement Learning](https://mlsanity.com/paper/2205.07802/) (2022)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks](https://mlsanity.com/paper/1806.05393/) (2018)
- [The Dormant Neuron Phenomenon in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.12902/) (2023)
- [Understanding plasticity in neural networks](https://mlsanity.com/paper/2303.01486/) (2023)
- [The Shattered Gradients Problem: If resnets are the answer, then what is the question?](https://mlsanity.com/paper/1702.08591/) (2017)
- [Human-Timescale Adaptation in an Open-Ended Task Space](https://mlsanity.com/paper/2301.07608/) (2023)
- [Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2010.14498/) (2020)
- [Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice](https://mlsanity.com/paper/1711.04735/) (2017)

## Built on by (most cited first, 21 of 21)

- [Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn](https://mlsanity.com/paper/2506.00592/) (2025)
- [MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2506.14990/) (2025)
- [FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoff](https://mlsanity.com/paper/2602.08040/) (2026)
- [Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning](https://mlsanity.com/paper/2509.22335/) (2025)
- [Rotation-Preserving Supervised Fine-Tuning](https://mlsanity.com/paper/2605.10973/) (2026)
- [Overcoming Non-stationary Dynamics with Evidential Proximal Policy Optimization](https://mlsanity.com/paper/2503.01468/) (2025)
- [Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learning](https://mlsanity.com/paper/2504.17490/) (2025)
- [Preserving Plasticity in Continual Learning with Adaptive Linearity Injection](https://mlsanity.com/paper/2505.09486/) (2025)
- [Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2508.06871/) (2025)
- [Do Neural Networks Lose Plasticity in a Gradually Changing World?](https://mlsanity.com/paper/2602.09234/) (2026)
- [The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning](https://mlsanity.com/paper/2604.01913/) (2026)
- [SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning](https://mlsanity.com/paper/2605.04712/) (2026)
- [Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning](https://mlsanity.com/paper/2510.12096/) (2025)
- [Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learning](https://mlsanity.com/paper/2602.20796/) (2026)
- [Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning](https://mlsanity.com/paper/2604.15414/) (2026)
- [Can Scale Save Us From Plasticity Loss in Large Language Models?](https://mlsanity.com/paper/2606.24752/) (2026)
- [ADARL: Adaptive Low-Rank Structures for Robust Policy Learning under Uncertainty](https://mlsanity.com/paper/2510.11899/) (2025)
- [Policy Gradient with Adaptive Entropy Annealing for Continual Fine-Tuning](https://mlsanity.com/paper/2602.14078/) (2026)
- [CDRL: A Reinforcement Learning Framework Inspired by Cerebellar Circuits and Dendritic Computational Strategies](https://mlsanity.com/paper/2602.15367/) (2026)
- [Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning](https://mlsanity.com/paper/2603.00903/) (2026)
- [SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learning](https://mlsanity.com/paper/2608.18319/) (2026)

## Abstract (arXiv)

Loss of plasticity, trainability loss, and primacy bias have been identified as issues arising when training deep neural networks on sequences of tasks -- all referring to the increased difficulty in training on new tasks. We propose to use Parseval regularization, which maintains orthogonality of weight matrices, to preserve useful optimization properties and improve training in a continual reinforcement learning setting. We show that it provides significant benefits to RL agents on a suite of gridworld, CARL and MetaWorld tasks. We conduct comprehensive ablations to identify the source of its benefits and investigate the effect of certain metrics associated to network trainability including weight matrix rank, weight norms and policy entropy.