# Deep Clustering with Features from Self-Supervised Pretraining

arXiv: https://arxiv.org/abs/2207.13364 · HTML: https://mlsanity.com/paper/2207.13364/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 27 Jul 2022.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering](https://mlsanity.com/paper/1610.04794/) (2016)
- [Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1611.02648/) (2016)
- [Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering](https://mlsanity.com/paper/1611.05148/) (2016)
- [Vision Transformers are Robust Learners](https://mlsanity.com/paper/2105.07581/) (2021)
- [SpectralNet: Spectral Clustering using Deep Neural Networks](https://mlsanity.com/paper/1801.01587/) (2018)
- [Deep Comprehensive Correlation Mining for Image Clustering](https://mlsanity.com/paper/1904.06925/) (2019)

## Built on by (most cited first, 14 of 14)

- [Exploring the Limits of Deep Image Clustering using Pretrained Models](https://mlsanity.com/paper/2303.17896/) (2023)
- [BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity](https://mlsanity.com/paper/2406.12723/) (2024)
- [Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models](https://mlsanity.com/paper/2306.05272/) (2023)
- [Self-supervised visual learning in the low-data regime: a comparative evaluation](https://mlsanity.com/paper/2404.17202/) (2024)
- [Rethinking cluster-conditioned diffusion models for label-free image synthesis](https://mlsanity.com/paper/2403.00570/) (2024)
- [An Empirical Study into Clustering of Unseen Datasets with Self-Supervised Encoders](https://mlsanity.com/paper/2406.02465/) (2024)
- [Clustering via Self-Supervised Diffusion](https://mlsanity.com/paper/2507.04283/) (2025)
- [Improving Image Clustering with Artifacts Attenuation via Inference-Time Attention Engineering](https://mlsanity.com/paper/2410.04801/) (2024)
- [Self-Enhanced Image Clustering with Cross-Modal Semantic Consistency](https://mlsanity.com/paper/2508.01254/) (2025)
- [Deep ContourFlow: Advancing Active Contours with Deep Learning](https://mlsanity.com/paper/2407.10696/) (2024)
- [Deep Clustering of Remote Sensing Scenes through Heterogeneous Transfer Learning](https://mlsanity.com/paper/2409.03938/) (2024)
- [Utilization of Neighbor Information for Image Classification with Different Levels of Supervision](https://mlsanity.com/paper/2503.14500/) (2025)
- [Unsupervised Image Classification with Adaptive Nearest Neighbor Selection and Cluster Ensembles](https://mlsanity.com/paper/2511.16213/) (2025)
- [Neighborhood Stability as a Measure of Nearest Neighbor Searchability](https://mlsanity.com/paper/2602.16673/) (2026)

## Abstract (arXiv)

A deep clustering model conceptually consists of a feature extractor that maps data points to a latent space, and a clustering head that groups data points into clusters in the latent space. Although the two components used to be trained jointly in an end-to-end fashion, recent works have proved it beneficial to train them separately in two stages. In the first stage, the feature extractor is trained via self-supervised learning, which enables the preservation of the cluster structures among the data points. To preserve the cluster structures even better, we propose to replace the first stage with another model that is pretrained on a much larger dataset via self-supervised learning. The method is simple and might suffer from domain shift. Nonetheless, we have empirically shown that it can achieve superior clustering performance. When a vision transformer (ViT) architecture is used for feature extraction, our method has achieved clustering accuracy 94.0%, 55.6% and 97.9% on CIFAR-10, CIFAR-100 and STL-10 respectively. The corresponding previous state-of-the-art results are 84.3%, 47.7% and 80.8%. Our code will be available online with the publication of the paper.