# DER: Dynamically Expandable Representation for Class Incremental Learning

arXiv: https://arxiv.org/abs/2103.16788 · HTML: https://mlsanity.com/paper/2103.16788/

## Key facts

- Cited by 355 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 31 Mar 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Overcoming Catastrophic Forgetting by Incremental Moment Matching](https://mlsanity.com/paper/1703.08475/) (2017)
- [Semantic Drift Compensation for Class-Incremental Learning](https://mlsanity.com/paper/2004.00440/) (2020)
- [Compacting, Picking and Growing for Unforgetting Continual Learning](https://mlsanity.com/paper/1910.06562/) (2019)
- [Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning](https://mlsanity.com/paper/1904.03137/) (2019)
- [Conditional Channel Gated Networks for Task-Aware Continual Learning](https://mlsanity.com/paper/2004.00070/) (2020)
- [Heated-Up Softmax Embedding](https://mlsanity.com/paper/1809.04157/) (2018)

## Built on by (most cited first, 30 of 355)

- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion](https://mlsanity.com/paper/2111.11326/) (2021)
- [S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning](https://mlsanity.com/paper/2207.12819/) (2022)
- [FOSTER: Feature Boosting and Compression for Class-Incremental Learning](https://mlsanity.com/paper/2204.04662/) (2022)
- [Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters](https://mlsanity.com/paper/2403.11549/) (2024)
- [Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need](https://mlsanity.com/paper/2303.07338/) (2023)
- [A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning](https://mlsanity.com/paper/2205.13218/) (2022)
- [Forward Compatible Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2203.06953/) (2022)
- [Federated Class-Incremental Learning](https://mlsanity.com/paper/2203.11473/) (2022)
- [Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2203.06359/) (2022)
- [Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation](https://mlsanity.com/paper/2204.00895/) (2022)
- [Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2303.06628/) (2023)
- [Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks](https://mlsanity.com/paper/2203.17030/) (2022)
- [FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning](https://mlsanity.com/paper/2309.14062/) (2023)
- [PyCIL: A Python Toolbox for Class-Incremental Learning](https://mlsanity.com/paper/2112.12533/) (2021)
- [A Theoretical Study on Solving Continual Learning](https://mlsanity.com/paper/2211.02633/) (2022)
- [Learning without Forgetting for Vision-Language Models](https://mlsanity.com/paper/2305.19270/) (2023)
- [Continual Learning and Catastrophic Forgetting](https://mlsanity.com/paper/2403.05175/) (2024)
- [AFEC: Active Forgetting of Negative Transfer in Continual Learning](https://mlsanity.com/paper/2110.12187/) (2021)
- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [MagMax: Leveraging Model Merging for Seamless Continual Learning](https://mlsanity.com/paper/2407.06322/) (2024)
- [CLIP model is an Efficient Continual Learner](https://mlsanity.com/paper/2210.03114/) (2022)
- [PILOT: A Pre-Trained Model-Based Continual Learning Toolbox](https://mlsanity.com/paper/2309.07117/) (2023)
- [Don't Stop Learning: Towards Continual Learning for the CLIP Model](https://mlsanity.com/paper/2207.09248/) (2022)
- [Computationally Budgeted Continual Learning: What Does Matter?](https://mlsanity.com/paper/2303.11165/) (2023)
- [Class-Incremental Learning with Strong Pre-trained Models](https://mlsanity.com/paper/2204.03634/) (2022)
- [ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection](https://mlsanity.com/paper/2205.14922/) (2022)

## Abstract (arXiv)

We address the problem of class incremental learning, which is a core step towards achieving adaptive vision intelligence. In particular, we consider the task setting of incremental learning with limited memory and aim to achieve better stability-plasticity trade-off. To this end, we propose a novel two-stage learning approach that utilizes a dynamically expandable representation for more effective incremental concept modeling. Specifically, at each incremental step, we freeze the previously learned representation and augment it with additional feature dimensions from a new learnable feature extractor. This enables us to integrate new visual concepts with retaining learned knowledge. We dynamically expand the representation according to the complexity of novel concepts by introducing a channel-level mask-based pruning strategy. Moreover, we introduce an auxiliary loss to encourage the model to learn diverse and discriminate features for novel concepts. We conduct extensive experiments on the three class incremental learning benchmarks and our method consistently outperforms other methods with a large margin.