# Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting

arXiv: https://arxiv.org/abs/1802.02950 · HTML: https://mlsanity.com/paper/1802.02950/

## Key facts

- Cited by 127 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2018.

## Built on by (most cited first, 30 of 127)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Rainbow Memory: Continual Learning with a Memory of Diverse Samples](https://mlsanity.com/paper/2103.17230/) (2021)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [Rehearsal revealed: The limits and merits of revisiting samples in continual learning](https://mlsanity.com/paper/2104.07446/) (2021)
- [Few-Shot Lifelong Learning](https://mlsanity.com/paper/2103.00991/) (2021)
- [ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection](https://mlsanity.com/paper/2205.14922/) (2022)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [Learning with Fantasy: Semantic-Aware Virtual Contrastive Constraint for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2304.00426/) (2023)
- [An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditions](https://mlsanity.com/paper/2204.08817/) (2022)
- [Continually Learning Self-Supervised Representations with Projected Functional Regularization](https://mlsanity.com/paper/2112.15022/) (2021)
- [Elastic Feature Consolidation for Cold Start Exemplar-Free Incremental Learning](https://mlsanity.com/paper/2402.03917/) (2024)
- [Long-Tailed Class Incremental Learning](https://mlsanity.com/paper/2210.00266/) (2022)
- [The Ideal Continual Learner: An Agent That Never Forgets](https://mlsanity.com/paper/2305.00316/) (2023)
- [Curriculum-Meta Learning for Order-Robust Continual Relation Extraction](https://mlsanity.com/paper/2101.01926/) (2021)
- [Open-world Machine Learning: Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2105.13448/) (2021)
- [Prototype-Guided Continual Adaptation for Class-Incremental Unsupervised Domain Adaptation](https://mlsanity.com/paper/2207.10856/) (2022)
- [Continual Learning via Sequential Function-Space Variational Inference](https://mlsanity.com/paper/2312.17210/) (2023)
- [Consistent Representation Learning for Continual Relation Extraction](https://mlsanity.com/paper/2203.02721/) (2022)
- [DS-AL: A Dual-Stream Analytic Learning for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2403.17503/) (2024)
- [CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning](https://mlsanity.com/paper/2505.24816/) (2025)
- [Towards Continual Egocentric Activity Recognition: A Multi-modal Egocentric Activity Dataset for Continual Learning](https://mlsanity.com/paper/2301.10931/) (2023)
- [OrCo: Towards Better Generalization via Orthogonality and Contrast for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2403.18550/) (2024)
- [ICICLE: Interpretable Class Incremental Continual Learning](https://mlsanity.com/paper/2303.07811/) (2023)
- [Fixed Design Analysis of Regularization-Based Continual Learning](https://mlsanity.com/paper/2303.10263/) (2023)
- [Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy](https://mlsanity.com/paper/2312.10549/) (2023)
- [Universal Representations: A Unified Look at Multiple Task and Domain Learning](https://mlsanity.com/paper/2204.02744/) (2022)
- [Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner](https://mlsanity.com/paper/2506.03595/) (2025)
- [Generative Multi-modal Models are Good Class-Incremental Learners](https://mlsanity.com/paper/2403.18383/) (2024)

## Abstract (arXiv)

In this paper we propose an approach to avoiding catastrophic forgetting in sequential task learning scenarios. Our technique is based on a network reparameterization that approximately diagonalizes the Fisher Information Matrix of the network parameters. This reparameterization takes the form of a factorized rotation of parameter space which, when used in conjunction with Elastic Weight Consolidation (which assumes a diagonal Fisher Information Matrix), leads to significantly better performance on lifelong learning of sequential tasks. Experimental results on the MNIST, CIFAR-100, CUB-200 and Stanford-40 datasets demonstrate that we significantly improve the results of standard elastic weight consolidation, and that we obtain competitive results when compared to other state-of-the-art in lifelong learning without forgetting.