# Prototype-Based Continual Learning with Label-free Replay Buffer and Cluster Preservation Loss

arXiv: https://arxiv.org/abs/2504.07240 · HTML: https://mlsanity.com/paper/2504.07240/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 9 Apr 2025.

## Builds on (most cited first, 22 of 22)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [CORe50: a New Dataset and Benchmark for Continuous Object Recognition](https://mlsanity.com/paper/1705.03550/) (2017)
- [New Insights on Reducing Abrupt Representation Change in Online Continual Learning](https://mlsanity.com/paper/2104.05025/) (2021)
- [SS-IL: Separated Softmax for Incremental Learning](https://mlsanity.com/paper/2003.13947/) (2020)
- [Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges](https://mlsanity.com/paper/1907.00182/) (2019)
- [Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworks](https://mlsanity.com/paper/2105.12374/) (2021)
- [Continual Learning on Dynamic Graphs via Parameter Isolation](https://mlsanity.com/paper/2305.13825/) (2023)
- [Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning](https://mlsanity.com/paper/2303.14771/) (2023)
- [Adaptive Memory Replay for Continual Learning](https://mlsanity.com/paper/2404.12526/) (2024)
- [PUDD: Towards Robust Multi-modal Prototype-based Deepfake Detection](https://mlsanity.com/paper/2406.15921/) (2024)
- [Dynamic Transformer Architecture for Continual Learning of Multimodal Tasks](https://mlsanity.com/paper/2401.15275/) (2024)
- [Self-Supervised Representation Learning for Adversarial Attack Detection](https://mlsanity.com/paper/2407.04382/) (2024)
- [Train-On-Request: An On-Device Continual Learning Workflow for Adaptive Real-World Brain Machine Interfaces](https://mlsanity.com/paper/2409.09161/) (2024)

## Built on by (most cited first, 9 of 9)

- [HiCL: Hippocampal-Inspired Continual Learning](https://mlsanity.com/paper/2508.16651/) (2025)
- [An Adaptive Resonance Theory-based Topological Clustering Algorithm with a Self-Adjusting Vigilance Parameter](https://mlsanity.com/paper/2511.17983/) (2025)
- [ProDER: A Continual Learning Approach for Fault Classification and Localization in Evolving Smart Grids](https://mlsanity.com/paper/2511.05420/) (2025)
- [Pushing the Limits of Distillation-Based Continual Learning via Classifier-Proximal Lightweight Plugins](https://mlsanity.com/paper/2512.03537/) (2025)
- [Adapting Actively on the Fly: Relevance-Guided Online Meta-Learning with Latent Concepts for Geospatial Discovery](https://mlsanity.com/paper/2602.17605/) (2026)
- [Unsupervised Continual Learning for Amortized Bayesian Inference](https://mlsanity.com/paper/2602.22884/) (2026)
- [Unsupervised Continual Clustering via Forward-Backward Knowledge Distillation](https://mlsanity.com/paper/2606.07474/) (2026)
- [Forgetting, plasticity, and co-observation: a third facet of continual learning](https://mlsanity.com/paper/2608.18803/) (2026)
- [Uncertainty-Aware Continual Learning for Open-World Intent Discovery Under an evolving Label Space](https://mlsanity.com/paper/2609.17866/) (2026)

## Abstract (arXiv)

Continual learning techniques employ simple replay sample selection processes and use them during subsequent tasks. Typically, they rely on labeled data. In this paper, we depart from this by automatically selecting prototypes stored without labels, preserving cluster structures in the latent space across tasks. By eliminating label dependence in the replay buffer and introducing cluster preservation loss, it is demonstrated that the proposed method can maintain essential information from previously encountered tasks while ensuring adaptation to new tasks. "Push-away" and "pull-toward" mechanisms over previously learned prototypes are also introduced for class-incremental and domain-incremental scenarios. These mechanisms ensure the retention of previously learned information as well as adaptation to new classes or domain shifts. The proposed method is evaluated on several benchmarks, including SplitCIFAR100, SplitImageNet32, SplitTinyImageNet, and SplitCaltech256 for class-incremental, as well as R-MNIST and CORe50 for domain-incremental setting using pre-extracted DINOv2 features. Experimental results indicate that the label-free replay-based technique outperforms state-of-the-art continual learning methods and, in some cases, even surpasses offline learning. An unsupervised variant of the proposed technique for the class-incremental setting, avoiding labels use even on incoming data, also demonstrated competitive performance, outperforming particular supervised baselines in some cases. These findings underscore the effectiveness of the proposed framework in retaining prior information and facilitating continual adaptation.