# Generative Feature Replay with Orthogonal Weight Modification for Continual Learning

arXiv: https://arxiv.org/abs/2005.03490 · HTML: https://mlsanity.com/paper/2005.03490/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2020.

## Built on by (most cited first, 10 of 10)

- [Continual Learning with Recursive Gradient Optimization](https://mlsanity.com/paper/2201.12522/) (2022)
- [Human-inspired Perspectives: A Survey on AI Long-term Memory](https://mlsanity.com/paper/2411.00489/) (2024)
- [False Memory Formation in Continual Learners Through Imperceptible Backdoor Trigger](https://mlsanity.com/paper/2202.04479/) (2022)
- [Tuned Compositional Feature Replays for Efficient Stream Learning](https://mlsanity.com/paper/2104.02206/) (2021)
- [Generative Feature-driven Image Replay for Continual Learning](https://mlsanity.com/paper/2106.05350/) (2021)
- [Unveiling the Tapestry: the Interplay of Generalization and Forgetting in Continual Learning](https://mlsanity.com/paper/2211.11174/) (2022)
- [Adversary Aware Continual Learning](https://mlsanity.com/paper/2304.14483/) (2023)
- [Continual Learners are Incremental Model Generalizers](https://mlsanity.com/paper/2306.12026/) (2023)
- [Overcoming Domain Drift in Online Continual Learning](https://mlsanity.com/paper/2405.09133/) (2024)
- [Defeating Catastrophic Forgetting via Enhanced Orthogonal Weights Modification](https://mlsanity.com/paper/2111.10078/) (2021)

## Abstract (arXiv)

The ability of intelligent agents to learn and remember multiple tasks sequentially is crucial to achieving artificial general intelligence. Many continual learning (CL) methods have been proposed to overcome catastrophic forgetting which results from non i.i.d data in the sequential learning of neural networks. In this paper we focus on class incremental learning, a challenging CL scenario. For this scenario, generative replay is a promising strategy which generates and replays pseudo data for previous tasks to alleviate catastrophic forgetting. However, it is hard to train a generative model continually for relatively complex data. Based on recently proposed orthogonal weight modification (OWM) algorithm which can approximately keep previously learned feature invariant when learning new tasks, we propose to 1) replay penultimate layer feature with a generative model; 2) leverage a self-supervised auxiliary task to further enhance the stability of feature. Empirical results on several datasets show our method always achieves substantial improvement over powerful OWM while conventional generative replay always results in a negative effect. Meanwhile our method beats several strong baselines including one based on real data storage. In addition, we conduct experiments to study why our method is effective.