# The information bottleneck and geometric clustering

arXiv: https://arxiv.org/abs/1712.09657 · HTML: https://mlsanity.com/paper/1712.09657/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Dec 2017.

## Built on by (most cited first, 8 of 8)

- [Investigating Out-of-Distribution Generalization of GNNs: An Architecture Perspective](https://mlsanity.com/paper/2402.08228/) (2024)
- [Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality](https://mlsanity.com/paper/2208.03848/) (2022)
- [Perturbation Theory for the Information Bottleneck](https://mlsanity.com/paper/2105.13977/) (2021)
- [Generalized Information Bottleneck for Gaussian Variables](https://mlsanity.com/paper/2303.17762/) (2023)
- [A Deterministic Information Bottleneck Method for Clustering Mixed-Type Data](https://mlsanity.com/paper/2407.03389/) (2024)
- [Topic Identification in LLM Input-Output Pairs through the Lens of Information Bottleneck](https://mlsanity.com/paper/2509.03533/) (2025)
- [Directed Information γ-covering: An Information-Theoretic Framework for Context Engineering](https://mlsanity.com/paper/2510.00079/) (2025)
- [Pareto-optimal clustering with the primal deterministic information bottleneck](https://mlsanity.com/paper/2204.02489/) (2022)

## Abstract (arXiv)

The information bottleneck (IB) approach to clustering takes a joint distribution P!left(X,Yright) and maps the data X to cluster labels T which retain maximal information about Y (Tishby et al., 1999). This objective results in an algorithm that clusters data points based upon the similarity of their conditional distributions P!left(Ymid Xright). This is in contrast to classic "geometric clustering'' algorithms such as k-means and gaussian mixture models (GMMs) which take a set of observed data points left xiright i=1:N and cluster them based upon their geometric (typically Euclidean) distance from one another. Here, we show how to use the deterministic information bottleneck (DIB) (Strouse and Schwab, 2017), a variant of IB, to perform geometric clustering, by choosing cluster labels that preserve information about data point location on a smoothed dataset. We also introduce a novel method to choose the number of clusters, based on identifying solutions where the tradeoff between number of clusters used and spatial information preserved is strongest. We apply this approach to a variety of simple clustering problems, showing that DIB with our model selection procedure recovers the generative cluster labels. We also show that, in particular limits of our model parameters, clustering with DIB and IB is equivalent to k-means and EM fitting of a GMM with hard and soft assignments, respectively. Thus, clustering with (D)IB generalizes and provides an information-theoretic perspective on these classic algorithms.