Forward-Backward Knowledge Distillation for Continual Clustering
On the map: Knowledge distillation → Distillation for continual learning
Abstract (arXiv)
Unsupervised Continual Learning (UCL) is a burgeoning field in machine learning, focusing on enabling neural networks to sequentially learn tasks without explicit label information. Catastrophic Forgetting (CF), where models forget previously learned tasks upon learning new ones, poses a significant challenge in continual learning, especially in UCL, where labeled information of data is not accessible. CF mitigation strategies, such as knowledge distillation and replay buffers, often face memory inefficiency and privacy issues. Although current research in UCL has endeavored to refine data representations and address CF in streaming data contexts, there is a noticeable lack of algorithms specifically designed for unsupervised clustering. To fill this gap, in this paper, we introduce the concept of Unsupervised Continual Clustering (UCC). We propose Forward-Backward Knowledge Distillation for unsupervised Continual Clustering (FBCC) to counteract CF within the context of UCC. FBCC employs a single continual learner (the ``teacher'') with a cluster projector, along with multiple student models, to address the CF issue. The proposed method consists of two phases: Forward Knowledge Distillation, where the teacher learns new clusters while retaining knowledge from previous tasks with guidance from specialized student models, and Backward Knowledge Distillation, where a student model mimics the teacher's behavior to retain task-specific knowledge, aiding the teacher in subsequent tasks. FBCC marks a pioneering approach to UCC, demonstrating enhanced performance and memory efficiency in clustering across various tasks, outperforming the application of clustering algorithms to the latent space of state-of-the-art UCL algorithms.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Distillation for continual learning, one of 12 directions in Knowledge distillation. Builds on 36 of its 53 references that are in the map. First posted on arXiv 29 May 2024.
Builds on
36 of 53 references.
References · 36
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
- 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
- 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
- 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
- 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
- 2015Unsupervised Deep Embedding for Clustering Analysiscited by 767
- 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 455
- 2020Contrastive Clusteringcited by 265
- 2021Co2L: Contrastive Continual Learningcited by 244
- 2016Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clusteringcited by 237
- 2016Joint Unsupervised Learning of Deep Representations and Image Clusterscited by 205
24 more
- 2021Self-Supervised Models are Continual Learnerscited by 149
- 2019Continual Unsupervised Representation Learningcited by 147
- 2020Continual Prototype Evolution: Learning Online from Non-Stationary Data Streamscited by 144
- 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 141
- 2020A Neural Dirichlet Process Mixture Model for Task-Free Continual Learningcited by 134
- 2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 117
- 2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 112
- 2020Continual Learning with Node-Importance based Adaptive Group Sparse Regularizationcited by 105
- 2021Representational Continuity for Unsupervised Continual Learningcited by 95
- 2018Memory Replay GANs: learning to generate images from new categories without forgettingcited by 77
- 2021ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learningcited by 68
- 2017Lifelong Generative Modelingcited by 51
- 2018Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologiescited by 43
- 2023Incorporating Neuro-Inspired Adaptability for Continual Learning in Artificial Intelligencecited by 43
- 2024Divide and not forget: Ensemble of selectively trained experts in Continual Learningcited by 38
- 2019Overcoming Catastrophic Forgetting by Neuron-level Plasticity Controlcited by 36
- 2019Unsupervised Progressive Learning and the STAM Architecturecited by 30
- 2021Unsupervised Continual Learning via Self-Adaptive Deep Clustering Approachcited by 19
- 2022Continual Learning Beyond a Single Modelcited by 18
- 2022SCALE: Online Self-Supervised Lifelong Learning without Prior Knowledgecited by 16
- 2023Plasticity-Optimized Complementary Networks for Unsupervised Continual Learningcited by 14
- 2021Continual Contrastive Learning for Image Classificationcited by 11
- 2021Doubly Contrastive Deep Clusteringcited by 9
- 2022C3: Cross-instance guided Contrastive Clusteringcited by 6
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Unsupervised Continual Clustering via Forward-Backward Knowledge Distillation
- 2024A Unified Framework for Continual Learning and Unlearning
- 2024Densely Distilling Cumulative Knowledge for Continual Learning
- 2022A Closer Look at Rehearsal-Free Continual Learning
- 2021Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer
- 2023Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning
- 2026The Forgetting-Retention Dilemma: Certified Unlearning Theory in Continual Learning
- 2023CUCL: Codebook for Unsupervised Continual Learning