# FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning

arXiv: https://arxiv.org/abs/2309.14062 · HTML: https://mlsanity.com/paper/2309.14062/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 80 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 30 of 60)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [Meta-learning with differentiable closed-form solvers](https://mlsanity.com/paper/1805.08136/) (2018)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples](https://mlsanity.com/paper/1903.03096/) (2019)
- [CORe50: a New Dataset and Benchmark for Continuous Object Recognition](https://mlsanity.com/paper/1705.03550/) (2017)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Meta Networks](https://mlsanity.com/paper/1703.00837/) (2017)
- [Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2004.10956/) (2020)

## Built on by (most cited first, 30 of 87)

- [Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2403.12030/) (2024)
- [Resurrecting Old Classes with New Data for Exemplar-Free Continual Learning](https://mlsanity.com/paper/2405.19074/) (2024)
- [Exemplar-free Continual Representation Learning via Learnable Drift Compensation](https://mlsanity.com/paper/2407.08536/) (2024)
- [A Practitioner's Guide to Continual Multimodal Pretraining](https://mlsanity.com/paper/2408.14471/) (2024)
- [Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learning](https://mlsanity.com/paper/2508.08165/) (2025)
- [LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning](https://mlsanity.com/paper/2503.18985/) (2025)
- [Few-Shot Class Incremental Learning with Attention-Aware Self-Adaptive Prompt](https://mlsanity.com/paper/2403.09857/) (2024)
- [Task-recency bias strikes back: Adapting covariances in Exemplar-Free Class Incremental Learning](https://mlsanity.com/paper/2409.18265/) (2024)
- [Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2509.16738/) (2025)
- [Random Representations Outperform Online Continually Learned Representations](https://mlsanity.com/paper/2402.08823/) (2024)
- [iNeMo: Incremental Neural Mesh Models for Robust Class-Incremental Learning](https://mlsanity.com/paper/2407.09271/) (2024)
- [Calibrating Higher-Order Statistics for Few-Shot Class-Incremental Learning with Pre-trained Vision Transformers](https://mlsanity.com/paper/2404.06622/) (2024)
- [Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering](https://mlsanity.com/paper/2502.04469/) (2025)
- [AIR: Analytic Imbalance Rectifier for Continual Learning](https://mlsanity.com/paper/2408.10349/) (2024)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [Order-Robust Class Incremental Learning: Graph-Driven Dynamic Similarity Grouping](https://mlsanity.com/paper/2502.20032/) (2025)
- [AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projection](https://mlsanity.com/paper/2511.13880/) (2025)
- [REAL: Representation Enhanced Analytic Learning for Exemplar-free Class-incremental Learning](https://mlsanity.com/paper/2403.13522/) (2024)
- [An Empirical Analysis of Forgetting in Pre-trained Models with Incremental Low-Rank Updates](https://mlsanity.com/paper/2405.18069/) (2024)
- [CEAT: Continual Expansion and Absorption Transformer for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2403.06670/) (2024)
- [How green is continual learning, really? Analyzing the energy consumption in continual training of vision foundation models](https://mlsanity.com/paper/2409.18664/) (2024)
- [Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Models](https://mlsanity.com/paper/2506.00037/) (2025)
- [HyperMask: Adaptive Hypernetwork-based Masks for Continual Learning](https://mlsanity.com/paper/2310.00113/) (2023)
- [Future-Proofing Class-Incremental Learning](https://mlsanity.com/paper/2404.03200/) (2024)
- [Exploring the Stability Gap in Continual Learning: The Role of the Classification Head](https://mlsanity.com/paper/2411.04723/) (2024)
- [Efficient LLM Moderation with Multi-Layer Latent Prototypes](https://mlsanity.com/paper/2502.16174/) (2025)
- [Reducing Class-wise Confusion for Incremental Learning with Disentangled Manifolds](https://mlsanity.com/paper/2503.17677/) (2025)
- [KAC: Kolmogorov-Arnold Classifier for Continual Learning](https://mlsanity.com/paper/2503.21076/) (2025)
- [AnalyticKWS: Towards Exemplar-Free Analytic Class Incremental Learning for Small-footprint Keyword Spotting](https://mlsanity.com/paper/2505.11817/) (2025)
- [Parameter Efficient Continual Learning with Dynamic Low-Rank Adaptation](https://mlsanity.com/paper/2505.11998/) (2025)

## Abstract (arXiv)

Exemplar-free class-incremental learning (CIL) poses several challenges since it prohibits the rehearsal of data from previous tasks and thus suffers from catastrophic forgetting. Recent approaches to incrementally learning the classifier by freezing the feature extractor after the first task have gained much attention. In this paper, we explore prototypical networks for CIL, which generate new class prototypes using the frozen feature extractor and classify the features based on the Euclidean distance to the prototypes. In an analysis of the feature distributions of classes, we show that classification based on Euclidean metrics is successful for jointly trained features. However, when learning from non-stationary data, we observe that the Euclidean metric is suboptimal and that feature distributions are heterogeneous. To address this challenge, we revisit the anisotropic Mahalanobis distance for CIL. In addition, we empirically show that modeling the feature covariance relations is better than previous attempts at sampling features from normal distributions and training a linear classifier. Unlike existing methods, our approach generalizes to both many- and few-shot CIL settings, as well as to domain-incremental settings. Interestingly, without updating the backbone network, our method obtains state-of-the-art results on several standard continual learning benchmarks. Code is available at https://github.com/dipamgoswami/FeCAM.