# Memory-Efficient Incremental Learning Through Feature Adaptation

arXiv: https://arxiv.org/abs/2004.00713 · HTML: https://mlsanity.com/paper/2004.00713/

## Key facts

- Cited by 107 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2020.

## Built on by (most cited first, 30 of 107)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion](https://mlsanity.com/paper/2111.11326/) (2021)
- [Rainbow Memory: Continual Learning with a Memory of Diverse Samples](https://mlsanity.com/paper/2103.17230/) (2021)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [FOSTER: Feature Boosting and Compression for Class-Incremental Learning](https://mlsanity.com/paper/2204.04662/) (2022)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need](https://mlsanity.com/paper/2303.07338/) (2023)
- [A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning](https://mlsanity.com/paper/2205.13218/) (2022)
- [Self-Supervised Models are Continual Learners](https://mlsanity.com/paper/2112.04215/) (2021)
- [Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks](https://mlsanity.com/paper/2203.17030/) (2022)
- [On Learning the Geodesic Path for Incremental Learning](https://mlsanity.com/paper/2104.08572/) (2021)
- [Learning without Forgetting for Vision-Language Models](https://mlsanity.com/paper/2305.19270/) (2023)
- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)
- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [RMM: Reinforced Memory Management for Class-Incremental Learning](https://mlsanity.com/paper/2301.05792/) (2023)
- [Essentials for Class Incremental Learning](https://mlsanity.com/paper/2102.09517/) (2021)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [Co-Transport for Class-Incremental Learning](https://mlsanity.com/paper/2107.12654/) (2021)
- [Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation](https://mlsanity.com/paper/2203.14098/) (2022)
- [How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?](https://mlsanity.com/paper/2410.17594/) (2024)
- [Exemplar-free Continual Representation Learning via Learnable Drift Compensation](https://mlsanity.com/paper/2407.08536/) (2024)
- [Class-Balanced Distillation for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2104.05279/) (2021)
- [Continual Learning with Bayesian Model based on a Fixed Pre-trained Feature Extractor](https://mlsanity.com/paper/2204.13349/) (2022)
- [Optimizing Reusable Knowledge for Continual Learning via Metalearning](https://mlsanity.com/paper/2106.05390/) (2021)
- [ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation](https://mlsanity.com/paper/2210.06816/) (2022)
- [BiRT: Bio-inspired Replay in Vision Transformers for Continual Learning](https://mlsanity.com/paper/2305.04769/) (2023)
- [Continual Segment: Towards a Single, Unified and Accessible Continual Segmentation Model of 143 Whole-body Organs in CT Scans](https://mlsanity.com/paper/2302.00162/) (2023)
- [Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractors](https://mlsanity.com/paper/2104.13614/) (2021)
- [RAW-Adapter: Adapting Pre-trained Visual Model to Camera RAW Images](https://mlsanity.com/paper/2408.14802/) (2024)

## Abstract (arXiv)

We introduce an approach for incremental learning that preserves feature descriptors of training images from previously learned classes, instead of the images themselves, unlike most existing work. Keeping the much lower-dimensional feature embeddings of images reduces the memory footprint significantly. We assume that the model is updated incrementally for new classes as new data becomes available sequentially.This requires adapting the previously stored feature vectors to the updated feature space without having access to the corresponding original training images. Feature adaptation is learned with a multi-layer perceptron, which is trained on feature pairs corresponding to the outputs of the original and updated network on a training image. We validate experimentally that such a transformation generalizes well to the features of the previous set of classes, and maps features to a discriminative subspace in the feature space. As a result, the classifier is optimized jointly over new and old classes without requiring old class images. Experimental results show that our method achieves state-of-the-art classification accuracy in incremental learning benchmarks, while having at least an order of magnitude lower memory footprint compared to image-preserving strategies.