# Exploring the Limits of Deep Image Clustering using Pretrained Models

arXiv: https://arxiv.org/abs/2303.17896 · HTML: https://mlsanity.com/paper/2303.17896/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 51 references that are in the map.
- First posted on arXiv 31 Mar 2023.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [Masked Siamese Networks for Label-Efficient Learning](https://mlsanity.com/paper/2204.07141/) (2022)
- [Understanding self-supervised Learning Dynamics without Contrastive Pairs](https://mlsanity.com/paper/2102.06810/) (2021)
- [Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering](https://mlsanity.com/paper/1610.04794/) (2016)
- [How Well Do Self-Supervised Models Transfer?](https://mlsanity.com/paper/2011.13377/) (2020)

## Built on by (most cited first, 30 of 33)

- [Diffusion Models and Representation Learning: A Survey](https://mlsanity.com/paper/2407.00783/) (2024)
- [A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advances](https://mlsanity.com/paper/2505.17799/) (2025)
- [Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models](https://mlsanity.com/paper/2306.05272/) (2023)
- [MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Representations](https://mlsanity.com/paper/2402.10093/) (2024)
- [I-Con: A Unifying Framework for Representation Learning](https://mlsanity.com/paper/2504.16929/) (2025)
- [ELFS: Label-Free Coreset Selection with Proxy Training Dynamics](https://mlsanity.com/paper/2406.04273/) (2024)
- [Exploring a Principled Framework for Deep Subspace Clustering](https://mlsanity.com/paper/2503.17288/) (2025)
- [CEIR: Concept-based Explainable Image Representation Learning](https://mlsanity.com/paper/2312.10747/) (2023)
- [Rethinking cluster-conditioned diffusion models for label-free image synthesis](https://mlsanity.com/paper/2403.00570/) (2024)
- [Style-based Clustering of Visual Artworks and the Play of Neural Style-Representations](https://mlsanity.com/paper/2409.08245/) (2024)
- [From Logits to Hierarchies: Hierarchical Clustering made Simple](https://mlsanity.com/paper/2410.07858/) (2024)
- [TreeDiffusion: Hierarchical Generative Clustering for Conditional Diffusion](https://mlsanity.com/paper/2410.16910/) (2024)
- [Clustering via Self-Supervised Diffusion](https://mlsanity.com/paper/2507.04283/) (2025)
- [pNNCLR: Stochastic Pseudo Neighborhoods for Contrastive Learning based Unsupervised Representation Learning Problems](https://mlsanity.com/paper/2308.06983/) (2023)
- [Deep Online Probability Aggregation Clustering](https://mlsanity.com/paper/2407.05246/) (2024)
- [Improving Image Clustering with Artifacts Attenuation via Inference-Time Attention Engineering](https://mlsanity.com/paper/2410.04801/) (2024)
- [Scaling Up Deep Clustering Methods Beyond ImageNet-1K](https://mlsanity.com/paper/2406.01203/) (2024)
- [Text-Guided Alternative Image Clustering](https://mlsanity.com/paper/2406.18589/) (2024)
- [Keep It Light! Simplifying Image Clustering Via Text-Free Adapters](https://mlsanity.com/paper/2502.04226/) (2025)
- [Self-Enhanced Image Clustering with Cross-Modal Semantic Consistency](https://mlsanity.com/paper/2508.01254/) (2025)
- [Beyond I-Con: Exploring New Dimension of Distance Measures in Representation Learning](https://mlsanity.com/paper/2509.04734/) (2025)
- [Deep Clustering of Remote Sensing Scenes through Heterogeneous Transfer Learning](https://mlsanity.com/paper/2409.03938/) (2024)
- [ConstellationNet: Reinventing Spatial Clustering through GNNs](https://mlsanity.com/paper/2503.07643/) (2025)
- [Utilization of Neighbor Information for Image Classification with Different Levels of Supervision](https://mlsanity.com/paper/2503.14500/) (2025)
- [Dimensionally Reduced Open-World Clustering: DROWCULA](https://mlsanity.com/paper/2509.07184/) (2025)
- [An Adaptor for Triggering Semi-Supervised Learning to Out-of-Box Serve Deep Image Clustering](https://mlsanity.com/paper/2509.20976/) (2025)
- [ClusterMine: Robust Label-Free Visual Out-Of-Distribution Detection via Concept Mining from Text Corpora](https://mlsanity.com/paper/2511.07068/) (2025)
- [Unsupervised Image Classification with Adaptive Nearest Neighbor Selection and Cluster Ensembles](https://mlsanity.com/paper/2511.16213/) (2025)
- [Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers](https://mlsanity.com/paper/2603.24275/) (2026)
- [Label-Efficient Dataset Pruning via Semi-Supervised Pseudo-Labeling](https://mlsanity.com/paper/2605.23198/) (2026)

## Abstract (arXiv)

We present a general methodology that learns to classify images without labels by leveraging pretrained feature extractors. Our approach involves self-distillation training of clustering heads based on the fact that nearest neighbours in the pretrained feature space are likely to share the same label. We propose a novel objective that learns associations between image features by introducing a variant of pointwise mutual information together with instance weighting. We demonstrate that the proposed objective is able to attenuate the effect of false positive pairs while efficiently exploiting the structure in the pretrained feature space. As a result, we improve the clustering accuracy over k-means on 17 different pretrained models by 6.1% and 12.2% on ImageNet and CIFAR100, respectively. Finally, using self-supervised vision transformers, we achieve a clustering accuracy of 61.6% on ImageNet. The code is available at https://github.com/HHU-MMBS/TEMI-official-BMVC2023.