# Generalised Mutual Information for Discriminative Clustering

arXiv: https://arxiv.org/abs/2210.06300 · HTML: https://mlsanity.com/paper/2210.06300/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 66 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [Contrastive Clustering](https://mlsanity.com/paper/2009.09687/) (2020)
- [Joint Unsupervised Learning of Deep Representations and Image Clusters](https://mlsanity.com/paper/1604.03628/) (2016)
- [Generalization and Equilibrium in Generative Adversarial Nets (GANs)](https://mlsanity.com/paper/1703.00573/) (2017)
- [Learning Discrete Representations via Information Maximizing Self-Augmented Training](https://mlsanity.com/paper/1702.08720/) (2017)
- [Deep Clustering via Joint Convolutional Autoencoder Embedding and Relative Entropy Minimization](https://mlsanity.com/paper/1704.06327/) (2017)
- [Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation](https://mlsanity.com/paper/2106.00131/) (2021)
- [DeepDPM: Deep Clustering With an Unknown Number of Clusters](https://mlsanity.com/paper/2203.14309/) (2022)
- [Improving Unsupervised Image Clustering With Robust Learning](https://mlsanity.com/paper/2012.11150/) (2020)
- [Deep clustering: On the link between discriminative models and K-means](https://mlsanity.com/paper/1810.04246/) (2018)
- [Clustering by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/2107.11635/) (2021)

## Built on by (most cited first, 5 of 5)

- [Kernel KMeans clustering splits for end-to-end unsupervised decision trees](https://mlsanity.com/paper/2402.12232/) (2024)
- [Sparse and geometry-aware generalisation of the mutual information for joint discriminative clustering and feature selection](https://mlsanity.com/paper/2302.03391/) (2023)
- [A Tutorial on Discriminative Clustering and Mutual Information](https://mlsanity.com/paper/2505.04484/) (2025)
- [Information Maximization for Long-Tailed Semi-Supervised Domain Generalization](https://mlsanity.com/paper/2603.08434/) (2026)
- [Turtle shell clustering: A mixture approach to discriminative clustering with applications to flow cytometry and other data](https://mlsanity.com/paper/2604.23083/) (2026)

## Abstract (arXiv)

In the last decade, recent successes in deep clustering majorly involved the mutual information (MI) as an unsupervised objective for training neural networks with increasing regularisations. While the quality of the regularisations have been largely discussed for improvements, little attention has been dedicated to the relevance of MI as a clustering objective. In this paper, we first highlight how the maximisation of MI does not lead to satisfying clusters. We identified the Kullback-Leibler divergence as the main reason of this behaviour. Hence, we generalise the mutual information by changing its core distance, introducing the generalised mutual information (GEMINI): a set of metrics for unsupervised neural network training. Unlike MI, some GEMINIs do not require regularisations when training. Some of these metrics are geometry-aware thanks to distances or kernels in the data space. Finally, we highlight that GEMINIs can automatically select a relevant number of clusters, a property that has been little studied in deep clustering context where the number of clusters is a priori unknown.