# Less-forgetting Learning in Deep Neural Networks

arXiv: https://arxiv.org/abs/1607.00122 · HTML: https://mlsanity.com/paper/1607.00122/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2016.

## Built on by (most cited first, 30 of 97)

- [Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation](https://mlsanity.com/paper/2204.00895/) (2022)
- [Fine-tuning can cripple your foundation model; preserving features may be the solution](https://mlsanity.com/paper/2308.13320/) (2023)
- [A Theoretical Study on Solving Continual Learning](https://mlsanity.com/paper/2211.02633/) (2022)
- [ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection](https://mlsanity.com/paper/2205.14922/) (2022)
- [An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditions](https://mlsanity.com/paper/2204.08817/) (2022)
- [Layerwise Optimization by Gradient Decomposition for Continual Learning](https://mlsanity.com/paper/2105.07561/) (2021)
- [Efficient Feature Transformations for Discriminative and Generative Continual Learning](https://mlsanity.com/paper/2103.13558/) (2021)
- [Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning](https://mlsanity.com/paper/2303.09483/) (2023)
- [Multi-domain Learning for Updating Face Anti-spoofing Models](https://mlsanity.com/paper/2208.11148/) (2022)
- [Elastic Feature Consolidation for Cold Start Exemplar-Free Incremental Learning](https://mlsanity.com/paper/2402.03917/) (2024)
- [A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm](https://mlsanity.com/paper/2310.12244/) (2023)
- [Balancing Stability and Plasticity through Advanced Null Space in Continual Learning](https://mlsanity.com/paper/2207.12061/) (2022)
- [Meta-attention for ViT-backed Continual Learning](https://mlsanity.com/paper/2203.11684/) (2022)
- [Hierarchical Prototype Networks for Continual Graph Representation Learning](https://mlsanity.com/paper/2111.15422/) (2021)
- [Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation](https://mlsanity.com/paper/2211.06866/) (2022)
- [DS-AL: A Dual-Stream Analytic Learning for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2403.17503/) (2024)
- [Dynamic Language Models for Continuously Evolving Content](https://mlsanity.com/paper/2106.06297/) (2021)
- [VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Model](https://mlsanity.com/paper/2403.05346/) (2024)
- [CLeaR: An Adaptive Continual Learning Framework for Regression Tasks](https://mlsanity.com/paper/2101.00926/) (2021)
- [Task-Free Continual Learning via Online Discrepancy Distance Learning](https://mlsanity.com/paper/2210.06579/) (2022)
- [Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospects](https://mlsanity.com/paper/2312.17004/) (2023)
- [GACL: Exemplar-Free Generalized Analytic Continual Learning](https://mlsanity.com/paper/2403.15706/) (2024)
- [Continual Learning Based on OOD Detection and Task Masking](https://mlsanity.com/paper/2203.09450/) (2022)
- [ConStruct-VL: Data-Free Continual Structured VL Concepts Learning](https://mlsanity.com/paper/2211.09790/) (2022)
- [Long-Tailed Continual Learning For Visual Food Recognition](https://mlsanity.com/paper/2307.00183/) (2023)
- [On the importance of cross-task features for class-incremental learning](https://mlsanity.com/paper/2106.11930/) (2021)
- [Lifelong Teacher-Student Network Learning](https://mlsanity.com/paper/2107.04689/) (2021)
- [CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation](https://mlsanity.com/paper/2310.06368/) (2023)
- [Towards Robust Graph Incremental Learning on Evolving Graphs](https://mlsanity.com/paper/2402.12987/) (2024)
- [Lifelong Learning with Sketched Structural Regularization](https://mlsanity.com/paper/2104.08604/) (2021)

## Abstract (arXiv)

A catastrophic forgetting problem makes deep neural networks forget the previously learned information, when learning data collected in new environments, such as by different sensors or in different light conditions. This paper presents a new method for alleviating the catastrophic forgetting problem. Unlike previous research, our method does not use any information from the source domain. Surprisingly, our method is very effective to forget less of the information in the source domain, and we show the effectiveness of our method using several experiments. Furthermore, we observed that the forgetting problem occurs between mini-batches when performing general training processes using stochastic gradient descent methods, and this problem is one of the factors that degrades generalization performance of the network. We also try to solve this problem using the proposed method. Finally, we show our less-forgetting learning method is also helpful to improve the performance of deep neural networks in terms of recognition rates.