# Elastic Feature Consolidation for Cold Start Exemplar-Free Incremental Learning

arXiv: https://arxiv.org/abs/2402.03917 · HTML: https://mlsanity.com/paper/2402.03917/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 6 Feb 2024.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Semantic Drift Compensation for Class-Incremental Learning](https://mlsanity.com/paper/2004.00440/) (2020)
- [New Insights on Reducing Abrupt Representation Change in Online Continual Learning](https://mlsanity.com/paper/2104.05025/) (2021)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning](https://mlsanity.com/paper/2205.13218/) (2022)
- [Adaptive Aggregation Networks for Class-Incremental Learning](https://mlsanity.com/paper/2010.05063/) (2020)
- [Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning](https://mlsanity.com/paper/2106.09701/) (2021)
- [Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting](https://mlsanity.com/paper/1802.02950/) (2018)
- [FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2211.13131/) (2022)
- [Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2203.06359/) (2022)
- [Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation](https://mlsanity.com/paper/2204.00895/) (2022)
- [Less-forgetting Learning in Deep Neural Networks](https://mlsanity.com/paper/1607.00122/) (2016)
- [First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning](https://mlsanity.com/paper/2303.13199/) (2023)
- [DeeSIL: Deep-Shallow Incremental Learning](https://mlsanity.com/paper/1808.06396/) (2018)
- [A Comprehensive Empirical Evaluation on Online Continual Learning](https://mlsanity.com/paper/2308.10328/) (2023)
- [One Pass ImageNet](https://mlsanity.com/paper/2111.01956/) (2021)
- [Model-centric Data Manifold: the Data Through the Eyes of the Model](https://mlsanity.com/paper/2104.13289/) (2021)

## Built on by (most cited first, 30 of 37)

- [Exemplar-free Continual Representation Learning via Learnable Drift Compensation](https://mlsanity.com/paper/2407.08536/) (2024)
- [Task-recency bias strikes back: Adapting covariances in Exemplar-Free Class Incremental Learning](https://mlsanity.com/paper/2409.18265/) (2024)
- [Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective](https://mlsanity.com/paper/2506.00205/) (2025)
- [PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2407.14029/) (2024)
- [On the Discrimination and Consistency for Exemplar-Free Class Incremental Learning](https://mlsanity.com/paper/2501.15454/) (2025)
- [Online Curvature-Aware Replay: Leveraging mathbf2nd Order Information for Online Continual Learning](https://mlsanity.com/paper/2502.01866/) (2025)
- [Semantic Shift Estimation via Dual-Projection and Classifier Reconstruction for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2503.05423/) (2025)
- [An Empirical Analysis of Forgetting in Pre-trained Models with Incremental Low-Rank Updates](https://mlsanity.com/paper/2405.18069/) (2024)
- [How green is continual learning, really? Analyzing the energy consumption in continual training of vision foundation models](https://mlsanity.com/paper/2409.18664/) (2024)
- [Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Models](https://mlsanity.com/paper/2506.00037/) (2025)
- [Overcoming Catastrophic Forgetting in Federated Class-Incremental Learning via Federated Global Twin Generator](https://mlsanity.com/paper/2407.11078/) (2024)
- [Covariances for Free: Exploiting Mean Distributions for Training-free Federated Learning](https://mlsanity.com/paper/2412.14326/) (2024)
- [PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning](https://mlsanity.com/paper/2601.13020/) (2026)
- [Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2407.17120/) (2024)
- [Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers](https://mlsanity.com/paper/2403.07404/) (2024)
- [EFC++: Elastic Feature Consolidation with Prototype Re-balancing for Cold Start Exemplar-free Incremental Learning](https://mlsanity.com/paper/2503.10439/) (2025)
- [Achieving Deep Continual Learning via Evolution](https://mlsanity.com/paper/2502.06210/) (2025)
- [Analytic Drift Resister for Non-Exemplar Continual Graph Learning](https://mlsanity.com/paper/2604.02633/) (2026)
- [Continual Learning of Domain-Invariant Representations](https://mlsanity.com/paper/2605.15775/) (2026)
- [Two-Way Is Better Than One: Bidirectional Alignment with Cycle Consistency for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2606.05675/) (2026)
- [Slowing Down Forgetting in Continual Learning](https://mlsanity.com/paper/2411.06916/) (2024)
- [Less is More: Efficient Model Merging with Binary Task Switch](https://mlsanity.com/paper/2412.00054/) (2024)
- [FeNeC: Enhancing Continual Learning via Feature Clustering with Neighbor- or Logit-Based Classification](https://mlsanity.com/paper/2503.14301/) (2025)
- [Restoring Forgotten Knowledge in Non-Exemplar Class Incremental Learning through Test-Time Semantic Evolution](https://mlsanity.com/paper/2503.16793/) (2025)
- [Gradient-Free Continual Learning](https://mlsanity.com/paper/2504.01219/) (2025)
- [Prototype Augmented Hypernetworks for Continual Learning](https://mlsanity.com/paper/2505.07450/) (2025)
- [What is the role of memorization in Continual Learning?](https://mlsanity.com/paper/2505.17664/) (2025)
- [Exemplar-Free Continual Learning for State Space Models](https://mlsanity.com/paper/2505.18604/) (2025)
- [Closing the Oracle Gap: Increment Vector Transformation for Class Incremental Learning](https://mlsanity.com/paper/2509.21898/) (2025)
- [Teaching Prompts to Coordinate: Hierarchical Layer-Grouped Prompt Tuning for Continual Learning](https://mlsanity.com/paper/2511.12090/) (2025)

## Abstract (arXiv)

Exemplar-Free Class Incremental Learning (EFCIL) aims to learn from a sequence of tasks without having access to previous task data. In this paper, we consider the challenging Cold Start scenario in which insufficient data is available in the first task to learn a high-quality backbone. This is especially challenging for EFCIL since it requires high plasticity, which results in feature drift which is difficult to compensate for in the exemplar-free setting. To address this problem, we propose a simple and effective approach that consolidates feature representations by regularizing drift in directions highly relevant to previous tasks and employs prototypes to reduce task-recency bias. Our method, called Elastic Feature Consolidation (EFC), exploits a tractable second-order approximation of feature drift based on an Empirical Feature Matrix (EFM). The EFM induces a pseudo-metric in feature space which we use to regularize feature drift in important directions and to update Gaussian prototypes used in a novel asymmetric cross entropy loss which effectively balances prototype rehearsal with data from new tasks. Experimental results on CIFAR-100, Tiny-ImageNet, ImageNet-Subset and ImageNet-1K demonstrate that Elastic Feature Consolidation is better able to learn new tasks by maintaining model plasticity and significantly outperform the state-of-the-art.