# Unsupervised Continual Clustering via Forward-Backward Knowledge Distillation

arXiv: https://arxiv.org/abs/2606.07474 · HTML: https://mlsanity.com/paper/2606.07474/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Distillation for continual learning, one of 12 directions in Knowledge distillation.
- Builds on 36 of its 53 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Placed in

- Knowledge distillation / [Distillation for continual learning](https://mlsanity.com/distill/distillation-for-continual-learning/)

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Contrastive Clustering](https://mlsanity.com/paper/2009.09687/) (2020)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)
- [Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering](https://mlsanity.com/paper/1610.04794/) (2016)
- [Self-Supervised Models are Continual Learners](https://mlsanity.com/paper/2112.04215/) (2021)
- [Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning](https://mlsanity.com/paper/2103.13885/) (2021)
- [Online Coreset Selection for Rehearsal-based Continual Learning](https://mlsanity.com/paper/2106.01085/) (2021)
- [Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System](https://mlsanity.com/paper/2201.12604/) (2022)
- [Continual Learning with Node-Importance based Adaptive Group Sparse Regularization](https://mlsanity.com/paper/2003.13726/) (2020)
- [Representational Continuity for Unsupervised Continual Learning](https://mlsanity.com/paper/2110.06976/) (2021)
- [Memory Replay GANs: learning to generate images from new categories without forgetting](https://mlsanity.com/paper/1809.02058/) (2018)
- [ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning](https://mlsanity.com/paper/2101.00407/) (2021)
- [Twin Contrastive Learning for Online Clustering](https://mlsanity.com/paper/2210.11680/) (2022)
- [Lifelong Generative Modeling](https://mlsanity.com/paper/1705.09847/) (2017)
- [Incorporating Neuro-Inspired Adaptability for Continual Learning in Artificial Intelligence](https://mlsanity.com/paper/2308.14991/) (2023)
- [Divide and not forget: Ensemble of selectively trained experts in Continual Learning](https://mlsanity.com/paper/2401.10191/) (2024)
- [Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control](https://mlsanity.com/paper/1907.13322/) (2019)
- [Unsupervised Progressive Learning and the STAM Architecture](https://mlsanity.com/paper/1904.02021/) (2019)
- [Knowledge Distillation in Deep Learning and its Applications](https://mlsanity.com/paper/2007.09029/) (2020)
- [Unsupervised Continual Learning in Streaming Environments](https://mlsanity.com/paper/2109.09282/) (2021)
- [SCALE: Online Self-Supervised Lifelong Learning without Prior Knowledge](https://mlsanity.com/paper/2208.11266/) (2022)
- [Plasticity-Optimized Complementary Networks for Unsupervised Continual Learning](https://mlsanity.com/paper/2309.06086/) (2023)
- [Continual Contrastive Learning for Image Classification](https://mlsanity.com/paper/2107.01776/) (2021)
- [ProtoDepth: Unsupervised Continual Depth Completion with Prototypes](https://mlsanity.com/paper/2503.12745/) (2025)

## Abstract (arXiv)

Unsupervised Continual Learning (UCL) aims to enable neural networks to learn sequential tasks without labels or access to past data. A major challenge in this setting is Catastrophic Forgetting, where models forget previously learned tasks upon learning new ones. This challenge is amplified in UCL due to the absence of labels to guide learning and memory retention. Existing mitigation strategies, such as knowledge distillation and replay buffers, often raise memory and privacy concerns. Moreover, current UCL methods largely overlook clustering-specific objectives. To fill this gap, we introduce Unsupervised Continual Clustering (UCC) and propose Forward-Backward Knowledge Distillation for Continual Clustering (FBCC). FBCC employs a continual teacher network with a clustering projector and lightweight task-specific students. Through a dual-phase forward-backward distillation process, the teacher learns new clusters while preserving previously discovered cluster structure without storing past data. FBCC represents a pioneering approach to UCC, demonstrating improved clustering performance across sequential tasks. Experiments on four benchmark datasets demonstrate that FBCC consistently outperforms existing continual learning baselines in clustering accuracy while significantly reducing catastrophic forgetting.