# Forward-Backward Knowledge Distillation for Continual Clustering

arXiv: https://arxiv.org/abs/2405.19234 · HTML: https://mlsanity.com/paper/2405.19234/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for continual learning, one of 12 directions in Knowledge distillation.
- Builds on 36 of its 53 references that are in the map.
- First posted on arXiv 29 May 2024.

## Placed in

- Knowledge distillation / [Distillation for continual learning](https://mlsanity.com/distill/distillation-for-continual-learning/)

## Builds on (most cited first, 30 of 36)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Contrastive Clustering](https://mlsanity.com/paper/2009.09687/) (2020)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)
- [Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering](https://mlsanity.com/paper/1610.04794/) (2016)
- [Joint Unsupervised Learning of Deep Representations and Image Clusters](https://mlsanity.com/paper/1604.03628/) (2016)
- [Self-Supervised Models are Continual Learners](https://mlsanity.com/paper/2112.04215/) (2021)
- [Continual Unsupervised Representation Learning](https://mlsanity.com/paper/1910.14481/) (2019)
- [Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams](https://mlsanity.com/paper/2009.00919/) (2020)
- [Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning](https://mlsanity.com/paper/2103.13885/) (2021)
- [A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning](https://mlsanity.com/paper/2001.00689/) (2020)
- [Online Coreset Selection for Rehearsal-based Continual Learning](https://mlsanity.com/paper/2106.01085/) (2021)
- [Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System](https://mlsanity.com/paper/2201.12604/) (2022)
- [Continual Learning with Node-Importance based Adaptive Group Sparse Regularization](https://mlsanity.com/paper/2003.13726/) (2020)
- [Representational Continuity for Unsupervised Continual Learning](https://mlsanity.com/paper/2110.06976/) (2021)
- [Memory Replay GANs: learning to generate images from new categories without forgetting](https://mlsanity.com/paper/1809.02058/) (2018)
- [ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning](https://mlsanity.com/paper/2101.00407/) (2021)
- [Lifelong Generative Modeling](https://mlsanity.com/paper/1705.09847/) (2017)
- [Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies](https://mlsanity.com/paper/1808.06508/) (2018)
- [Incorporating Neuro-Inspired Adaptability for Continual Learning in Artificial Intelligence](https://mlsanity.com/paper/2308.14991/) (2023)
- [Divide and not forget: Ensemble of selectively trained experts in Continual Learning](https://mlsanity.com/paper/2401.10191/) (2024)
- [Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control](https://mlsanity.com/paper/1907.13322/) (2019)
- [Unsupervised Progressive Learning and the STAM Architecture](https://mlsanity.com/paper/1904.02021/) (2019)
- [Unsupervised Continual Learning via Self-Adaptive Deep Clustering Approach](https://mlsanity.com/paper/2106.14563/) (2021)

## Built on by (most cited first, 1 of 1)

- [An Adaptive Resonance Theory-based Topological Clustering Algorithm with a Self-Adjusting Vigilance Parameter](https://mlsanity.com/paper/2511.17983/) (2025)

## Abstract (arXiv)

Unsupervised Continual Learning (UCL) is a burgeoning field in machine learning, focusing on enabling neural networks to sequentially learn tasks without explicit label information. Catastrophic Forgetting (CF), where models forget previously learned tasks upon learning new ones, poses a significant challenge in continual learning, especially in UCL, where labeled information of data is not accessible. CF mitigation strategies, such as knowledge distillation and replay buffers, often face memory inefficiency and privacy issues. Although current research in UCL has endeavored to refine data representations and address CF in streaming data contexts, there is a noticeable lack of algorithms specifically designed for unsupervised clustering. To fill this gap, in this paper, we introduce the concept of Unsupervised Continual Clustering (UCC). We propose Forward-Backward Knowledge Distillation for unsupervised Continual Clustering (FBCC) to counteract CF within the context of UCC. FBCC employs a single continual learner (the ``teacher'') with a cluster projector, along with multiple student models, to address the CF issue. The proposed method consists of two phases: Forward Knowledge Distillation, where the teacher learns new clusters while retaining knowledge from previous tasks with guidance from specialized student models, and Backward Knowledge Distillation, where a student model mimics the teacher's behavior to retain task-specific knowledge, aiding the teacher in subsequent tasks. FBCC marks a pioneering approach to UCC, demonstrating enhanced performance and memory efficiency in clustering across various tasks, outperforming the application of clustering algorithms to the latent space of state-of-the-art UCL algorithms.