# IF2Net: Innately Forgetting-Free Networks for Continual Learning

arXiv: https://arxiv.org/abs/2306.10480 · HTML: https://mlsanity.com/paper/2306.10480/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 91 references that are in the map.
- First posted on arXiv 18 Jun 2023.

## Builds on (most cited first, 30 of 43)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)
- [Rainbow Memory: Continual Learning with a Memory of Diverse Samples](https://mlsanity.com/paper/2103.17230/) (2021)
- [Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines](https://mlsanity.com/paper/1810.12488/) (2018)
- [REMIND Your Neural Network to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/1910.02509/) (2019)
- [Continual Learning of Context-dependent Processing in Neural Networks](https://mlsanity.com/paper/1810.01256/) (2018)
- [Class-incremental Learning via Deep Model Consolidation](https://mlsanity.com/paper/1903.07864/) (2019)
- [Using Hindsight to Anchor Past Knowledge in Continual Learning](https://mlsanity.com/paper/2002.08165/) (2020)
- [Scalable and Order-robust Continual Learning with Additive Parameter Decomposition](https://mlsanity.com/paper/1902.09432/) (2019)
- [Continual Deep Learning by Functional Regularisation of Memorable Past](https://mlsanity.com/paper/2004.14070/) (2020)
- [Generative replay with feedback connections as a general strategy for continual learning](https://mlsanity.com/paper/1809.10635/) (2018)
- [Less-forgetting Learning in Deep Neural Networks](https://mlsanity.com/paper/1607.00122/) (2016)
- [Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges](https://mlsanity.com/paper/1907.00182/) (2019)
- [Lifelong GAN: Continual Learning for Conditional Image Generation](https://mlsanity.com/paper/1907.10107/) (2019)
- [Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning](https://mlsanity.com/paper/2110.04593/) (2021)
- [Class-Incremental Learning with Strong Pre-trained Models](https://mlsanity.com/paper/2204.03634/) (2022)

## Built on by (most cited first, 3 of 3)

- [Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding](https://mlsanity.com/paper/2401.09067/) (2024)
- [Incremental Online Learning of Randomized Neural Network with Forward Regularization](https://mlsanity.com/paper/2412.13096/) (2024)
- [The Sequential Price of Continual Learning](https://mlsanity.com/paper/2609.29674/) (2026)

## Abstract (arXiv)

Continual learning can incrementally absorb new concepts without interfering with previously learned knowledge. Motivated by the characteristics of neural networks, in which information is stored in weights on connections, we investigated how to design an Innately Forgetting-Free Network (IF2Net) for continual learning context. This study proposed a straightforward yet effective learning paradigm by ingeniously keeping the weights relative to each seen task untouched before and after learning a new task. We first presented the novel representation-level learning on task sequences with random weights. This technique refers to tweaking the drifted representations caused by randomization back to their separate task-optimal working states, but the involved weights are frozen and reused (opposite to well-known layer-wise updates of weights). Then, sequential decision-making without forgetting can be achieved by projecting the output weight updates into the parsimonious orthogonal space, making the adaptations not disturb old knowledge while maintaining model plasticity. IF2Net allows a single network to inherently learn unlimited mapping rules without telling task identities at test time by integrating the respective strengths of randomization and orthogonalization. We validated the effectiveness of our approach in the extensive theoretical analysis and empirical study.