# Mnemonics Training: Multi-Class Incremental Learning without Forgetting

arXiv: https://arxiv.org/abs/2002.10211 · HTML: https://mlsanity.com/paper/2002.10211/

## Key facts

- Cited by 221 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2020.

## Built on by (most cited first, 30 of 221)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Prompt-aligned Gradient for Prompt Tuning](https://mlsanity.com/paper/2205.14865/) (2022)
- [Rainbow Memory: Continual Learning with a Memory of Diverse Samples](https://mlsanity.com/paper/2103.17230/) (2021)
- [S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning](https://mlsanity.com/paper/2207.12819/) (2022)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Continual Pre-training of Language Models](https://mlsanity.com/paper/2302.03241/) (2023)
- [Few-Shot Incremental Learning with Continually Evolved Classifiers](https://mlsanity.com/paper/2104.03047/) (2021)
- [Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need](https://mlsanity.com/paper/2303.07338/) (2023)
- [Dataset Distillation using Neural Feature Regression](https://mlsanity.com/paper/2206.00719/) (2022)
- [A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning](https://mlsanity.com/paper/2205.13218/) (2022)
- [Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond](https://mlsanity.com/paper/2101.11517/) (2021)
- [Training Networks in Null Space of Feature Covariance for Continual Learning](https://mlsanity.com/paper/2103.07113/) (2021)
- [Federated Class-Incremental Learning](https://mlsanity.com/paper/2203.11473/) (2022)
- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)
- [FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2211.13131/) (2022)
- [Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima](https://mlsanity.com/paper/2111.01549/) (2021)
- [Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2203.06359/) (2022)
- [Distilling Causal Effect of Data in Class-Incremental Learning](https://mlsanity.com/paper/2103.01737/) (2021)
- [Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2403.12030/) (2024)
- [Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation](https://mlsanity.com/paper/2204.00895/) (2022)
- [Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2107.08918/) (2021)
- [Constrained Few-shot Class-incremental Learning](https://mlsanity.com/paper/2203.16588/) (2022)
- [Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning](https://mlsanity.com/paper/2112.02706/) (2021)
- [On Learning the Geodesic Path for Incremental Learning](https://mlsanity.com/paper/2104.08572/) (2021)
- [Memory Replay with Data Compression for Continual Learning](https://mlsanity.com/paper/2202.06592/) (2022)
- [A Theoretical Study on Solving Continual Learning](https://mlsanity.com/paper/2211.02633/) (2022)
- [Learning without Forgetting for Vision-Language Models](https://mlsanity.com/paper/2305.19270/) (2023)
- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)

## Abstract (arXiv)

Multi-Class Incremental Learning (MCIL) aims to learn new concepts by incrementally updating a model trained on previous concepts. However, there is an inherent trade-off to effectively learning new concepts without catastrophic forgetting of previous ones. To alleviate this issue, it has been proposed to keep around a few examples of the previous concepts but the effectiveness of this approach heavily depends on the representativeness of these examples. This paper proposes a novel and automatic framework we call mnemonics, where we parameterize exemplars and make them optimizable in an end-to-end manner. We train the framework through bilevel optimizations, i.e., model-level and exemplar-level. We conduct extensive experiments on three MCIL benchmarks, CIFAR-100, ImageNet-Subset and ImageNet, and show that using mnemonics exemplars can surpass the state-of-the-art by a large margin. Interestingly and quite intriguingly, the mnemonics exemplars tend to be on the boundaries between different classes.