# Unsupervised Visual Representation Learning by Online Constrained K-Means

arXiv: https://arxiv.org/abs/2105.11527 · HTML: https://mlsanity.com/paper/2105.11527/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 49 references that are in the map.
- First posted on arXiv 24 May 2021.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [No Fuss Distance Metric Learning using Proxies](https://mlsanity.com/paper/1703.07464/) (2017)
- [Local Aggregation for Unsupervised Learning of Visual Embeddings](https://mlsanity.com/paper/1903.12355/) (2019)
- [SoftTriple Loss: Deep Metric Learning Without Triplet Sampling](https://mlsanity.com/paper/1909.05235/) (2019)

## Built on by (most cited first, 27 of 27)

- [Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training](https://mlsanity.com/paper/2305.13689/) (2023)
- [Unicom: Universal and Compact Representation Learning for Image Retrieval](https://mlsanity.com/paper/2304.05884/) (2023)
- [Stable Cluster Discrimination for Deep Clustering](https://mlsanity.com/paper/2311.14310/) (2023)
- [Unsupervised Visual Representation Learning by Synchronous Momentum Grouping](https://mlsanity.com/paper/2207.06167/) (2022)
- [Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering](https://mlsanity.com/paper/2404.15655/) (2024)
- [CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination](https://mlsanity.com/paper/2408.09441/) (2024)
- [Multi-label Cluster Discrimination for Visual Representation Learning](https://mlsanity.com/paper/2407.17331/) (2024)
- [ProtoCon: Pseudo-label Refinement via Online Clustering and Prototypical Consistency for Efficient Semi-supervised Learning](https://mlsanity.com/paper/2303.13556/) (2023)
- [Contextually Affinitive Neighborhood Refinery for Deep Clustering](https://mlsanity.com/paper/2312.07806/) (2023)
- [End-to-end Learnable Clustering for Intent Learning in Recommendation](https://mlsanity.com/paper/2401.05975/) (2024)
- [EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Models](https://mlsanity.com/paper/2410.07133/) (2024)
- [Revisiting Foreground and Background Separation in Weakly-supervised Temporal Action Localization: A Clustering-based Approach](https://mlsanity.com/paper/2312.14138/) (2023)
- [Improved Visual Fine-tuning with Natural Language Supervision](https://mlsanity.com/paper/2304.01489/) (2023)
- [Model-Based Reinforcement Learning with Isolated Imaginations](https://mlsanity.com/paper/2303.14889/) (2023)
- [Dual-disentangled Deep Multiple Clustering](https://mlsanity.com/paper/2402.05310/) (2024)
- [Self-Supervised Learning for Text Recognition: A Critical Survey](https://mlsanity.com/paper/2407.19889/) (2024)
- [Representation Learning via Consistent Assignment of Views over Random Partitions](https://mlsanity.com/paper/2310.12692/) (2023)
- [Deep Online Probability Aggregation Clustering](https://mlsanity.com/paper/2407.05246/) (2024)
- [How PARTs assemble into wholes: Learning the relative composition of images](https://mlsanity.com/paper/2506.03682/) (2025)
- [Cluster Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2507.12359/) (2025)
- [Efficient Unsupervised Visual Representation Learning with Explicit Cluster Balancing](https://mlsanity.com/paper/2407.11168/) (2024)
- [Agent-Centric Personalized Multiple Clustering with Multi-Modal LLMs](https://mlsanity.com/paper/2503.22241/) (2025)
- [Reconstruction Task Finds Universal Winning Tickets](https://mlsanity.com/paper/2202.11484/) (2022)
- [SeA: Semantic Adversarial Augmentation for Last Layer Features from Unsupervised Representation Learning](https://mlsanity.com/paper/2408.13351/) (2024)
- [Multi-modal Dynamic Proxy Learning for Personalized Multiple Clustering](https://mlsanity.com/paper/2511.07274/) (2025)
- [Unsupervised Image Classification with Adaptive Nearest Neighbor Selection and Cluster Ensembles](https://mlsanity.com/paper/2511.16213/) (2025)
- [Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Information](https://mlsanity.com/paper/2608.08418/) (2026)

## Abstract (arXiv)

Cluster discrimination is an effective pretext task for unsupervised representation learning, which often consists of two phases: clustering and discrimination. Clustering is to assign each instance a pseudo label that will be used to learn representations in discrimination. The main challenge resides in clustering since prevalent clustering methods (e.g., k-means) have to run in a batch mode. Besides, there can be a trivial solution consisting of a dominating cluster. To address these challenges, we first investigate the objective of clustering-based representation learning. Based on this, we propose a novel clustering-based pretext task with online Constrained K-means (CoKe). Compared with the balanced clustering that each cluster has exactly the same size, we only constrain the minimal size of each cluster to flexibly capture the inherent data structure. More importantly, our online assignment method has a theoretical guarantee to approach the global optimum. By decoupling clustering and discrimination, CoKe can achieve competitive performance when optimizing with only a single view from each instance. Extensive experiments on ImageNet and other benchmark data sets verify both the efficacy and efficiency of our proposal. Code is available at urlhttps://github.com/idstcv/CoKe.