# Clustering by Maximizing Mutual Information Across Views

arXiv: https://arxiv.org/abs/2107.11635 · HTML: https://mlsanity.com/paper/2107.11635/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 55 references that are in the map.
- First posted on arXiv 24 Jul 2021.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring](https://mlsanity.com/paper/1911.09785/) (2019)
- [Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks](https://mlsanity.com/paper/1406.6909/) (2014)
- [Contrastive Clustering](https://mlsanity.com/paper/2009.09687/) (2020)
- [Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering](https://mlsanity.com/paper/1610.04794/) (2016)
- [Unsupervised Embedding Learning via Invariant and Spreading Instance Feature](https://mlsanity.com/paper/1904.03436/) (2019)
- [Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering](https://mlsanity.com/paper/1611.05148/) (2016)

## Built on by (most cited first, 23 of 23)

- [A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directions](https://mlsanity.com/paper/2206.07579/) (2022)
- [Neural Manifold Clustering and Embedding](https://mlsanity.com/paper/2201.10000/) (2022)
- [Multimodal Variational Auto-encoder based Audio-Visual Segmentation](https://mlsanity.com/paper/2310.08303/) (2023)
- [Heterogeneous Contrastive Learning for Foundation Models and Beyond](https://mlsanity.com/paper/2404.00225/) (2024)
- [DLME: Deep Local-flatness Manifold Embedding](https://mlsanity.com/paper/2207.03160/) (2022)
- [Joint Self-Supervised Image-Volume Representation Learning with Intra-Inter Contrastive Clustering](https://mlsanity.com/paper/2212.01893/) (2022)
- [DeNVeR: Deformable Neural Vessel Representations for Unsupervised Video Vessel Segmentation](https://mlsanity.com/paper/2406.01591/) (2024)
- [Self-supervised Context-aware Style Representation for Expressive Speech Synthesis](https://mlsanity.com/paper/2206.12559/) (2022)
- [On the Provable Importance of Gradients for Language-Assisted Image Clustering](https://mlsanity.com/paper/2510.16335/) (2025)
- [Identifying Backdoored Graphs in Graph Neural Network Training: An Explanation-Based Approach with Novel Metrics](https://mlsanity.com/paper/2403.18136/) (2024)
- [Generalised Mutual Information for Discriminative Clustering](https://mlsanity.com/paper/2210.06300/) (2022)
- [Branch-Tuning: Balancing Stability and Plasticity for Continual Self-Supervised Learning](https://mlsanity.com/paper/2403.18266/) (2024)
- [Convex Combination Consistency between Neighbors for Weakly-supervised Action Localization](https://mlsanity.com/paper/2205.00400/) (2022)
- [Learning Generalized Hybrid Proximity Representation for Image Recognition](https://mlsanity.com/paper/2301.13459/) (2023)
- [A Prototype-Oriented Clustering for Domain Shift with Source Privacy](https://mlsanity.com/paper/2302.03807/) (2023)
- [FreeCOS: Self-Supervised Learning from Fractals and Unlabeled Images for Curvilinear Object Segmentation](https://mlsanity.com/paper/2307.07245/) (2023)
- [Generalised Mutual Information: a Framework for Discriminative Clustering](https://mlsanity.com/paper/2309.02858/) (2023)
- [Aligning the Query Space: Greedy Information Projection for Language Model Data Selection](https://mlsanity.com/paper/2603.13790/) (2026)
- [CLUBench: A Clustering Benchmark](https://mlsanity.com/paper/2605.29933/) (2026)
- [Grid Jigsaw Representation with CLIP: A New Perspective on Image Clustering](https://mlsanity.com/paper/2310.17869/) (2023)
- [Deep Clustering via Gradual Community Detection](https://mlsanity.com/paper/2501.02036/) (2025)
- [A Tutorial on Discriminative Clustering and Mutual Information](https://mlsanity.com/paper/2505.04484/) (2025)
- [Reducing Bias and Variance: Generative Semantic Guidance and Bi-Layer Ensemble for Image Clustering](https://mlsanity.com/paper/2605.12961/) (2026)

## Abstract (arXiv)

We propose a novel framework for image clustering that incorporates joint representation learning and clustering. Our method consists of two heads that share the same backbone network - a "representation learning" head and a "clustering" head. The "representation learning" head captures fine-grained patterns of objects at the instance level which serve as clues for the "clustering" head to extract coarse-grain information that separates objects into clusters. The whole model is trained in an end-to-end manner by minimizing the weighted sum of two sample-oriented contrastive losses applied to the outputs of the two heads. To ensure that the contrastive loss corresponding to the "clustering" head is optimal, we introduce a novel critic function called "log-of-dot-product". Extensive experimental results demonstrate that our method significantly outperforms state-of-the-art single-stage clustering methods across a variety of image datasets, improving over the best baseline by about 5-7% in accuracy on CIFAR10/20, STL10, and ImageNet-Dogs. Further, the "two-stage" variant of our method also achieves better results than baselines on three challenging ImageNet subsets.