# Learning Invariant Representation for Continual Learning

arXiv: https://arxiv.org/abs/2101.06162 · HTML: https://mlsanity.com/paper/2101.06162/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 15 Jan 2021.

## Builds on (most cited first, 24 of 24)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [Adversarial Autoencoders](https://mlsanity.com/paper/1511.05644/) (2015)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [cGANs with Projection Discriminator](https://mlsanity.com/paper/1802.05637/) (2018)
- [Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines](https://mlsanity.com/paper/1810.12488/) (2018)
- [Towards Robust Evaluations of Continual Learning](https://mlsanity.com/paper/1805.09733/) (2018)
- [Adversarial Continual Learning](https://mlsanity.com/paper/2003.09553/) (2020)
- [SpaceNet: Make Free Space For Continual Learning](https://mlsanity.com/paper/2007.07617/) (2020)
- [Auto-Embedding Generative Adversarial Networks for High Resolution Image Synthesis](https://mlsanity.com/paper/1903.11250/) (2019)
- [Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data](https://mlsanity.com/paper/1610.05555/) (2016)
- [A Cyclically-Trained Adversarial Network for Invariant Representation Learning](https://mlsanity.com/paper/1906.09313/) (2019)

## Built on by (most cited first, 6 of 6)

- [Adversarial Continual Learning for Multi-Domain Hippocampal Segmentation](https://mlsanity.com/paper/2107.08751/) (2021)
- [Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning](https://mlsanity.com/paper/2409.19245/) (2024)
- [Self-Attention Meta-Learner for Continual Learning](https://mlsanity.com/paper/2101.12136/) (2021)
- [Avoiding Forgetting and Allowing Forward Transfer in Continual Learning via Sparse Networks](https://mlsanity.com/paper/2110.05329/) (2021)
- [Unsupervised Unlearning of Concept Drift with Autoencoders](https://mlsanity.com/paper/2211.12989/) (2022)
- [Continual Learning With Quasi-Newton Methods](https://mlsanity.com/paper/2503.19939/) (2025)

## Abstract (arXiv)

Continual learning aims to provide intelligent agents that are capable of learning continually a sequence of tasks, building on previously learned knowledge. A key challenge in this learning paradigm is catastrophically forgetting previously learned tasks when the agent faces a new one. Current rehearsal-based methods show their success in mitigating the catastrophic forgetting problem by replaying samples from previous tasks during learning a new one. However, these methods are infeasible when the data of previous tasks is not accessible. In this work, we propose a new pseudo-rehearsal-based method, named learning Invariant Representation for Continual Learning (IRCL), in which class-invariant representation is disentangled from a conditional generative model and jointly used with class-specific representation to learn the sequential tasks. Disentangling the shared invariant representation helps to learn continually a sequence of tasks, while being more robust to forgetting and having better knowledge transfer. We focus on class incremental learning where there is no knowledge about task identity during inference. We empirically evaluate our proposed method on two well-known benchmarks for continual learning: split MNIST and split Fashion MNIST. The experimental results show that our proposed method outperforms regularization-based methods by a big margin and is better than the state-of-the-art pseudo-rehearsal-based method. Finally, we analyze the role of the shared invariant representation in mitigating the forgetting problem especially when the number of replayed samples for each previous task is small.