# Phase Transitions for the Information Bottleneck in Representation Learning

arXiv: https://arxiv.org/abs/2001.01878 · HTML: https://mlsanity.com/paper/2001.01878/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2020.

## Built on by (most cited first, 11 of 11)

- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [Learning to Accelerate Partial Differential Equations via Latent Global Evolution](https://mlsanity.com/paper/2206.07681/) (2022)
- [Leveraging Information Bottleneck for Scientific Document Summarization](https://mlsanity.com/paper/2110.01280/) (2021)
- [Consistent and Asymptotically Unbiased Estimation of Proper Calibration Errors](https://mlsanity.com/paper/2312.08589/) (2023)
- [Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality](https://mlsanity.com/paper/2208.03848/) (2022)
- [Perturbation Theory for the Information Bottleneck](https://mlsanity.com/paper/2105.13977/) (2021)
- [A Provably Convergent Information Bottleneck Solution via ADMM](https://mlsanity.com/paper/2102.04729/) (2021)
- [Pareto-optimal clustering with the primal deterministic information bottleneck](https://mlsanity.com/paper/2204.02489/) (2022)
- [The Information Bottleneck's Ordinary Differential Equation: First-Order Root-Tracking for the IB](https://mlsanity.com/paper/2306.09790/) (2023)
- [Phase Transitions between Accuracy Regimes in L2 regularized Deep Neural Networks](https://mlsanity.com/paper/2505.06597/) (2025)
- [Stable and Convexified Information Bottleneck Optimization via Symbolic Continuation and Entropy-Regularized Trajectories](https://mlsanity.com/paper/2505.09239/) (2025)

## Abstract (arXiv)

In the Information Bottleneck (IB), when tuning the relative strength between compression and prediction terms, how do the two terms behave, and what's their relationship with the dataset and the learned representation? In this paper, we set out to answer these questions by studying multiple phase transitions in the IB objective: IBβ[p(z|x)] = I(X; Z) - β I(Y; Z) defined on the encoding distribution p(z|x) for input X, target Y and representation Z, where sudden jumps of dI(Y; Z)/d β and prediction accuracy are observed with increasing β. We introduce a definition for IB phase transitions as a qualitative change of the IB loss landscape, and show that the transitions correspond to the onset of learning new classes. Using second-order calculus of variations, we derive a formula that provides a practical condition for IB phase transitions, and draw its connection with the Fisher information matrix for parameterized models. We provide two perspectives to understand the formula, revealing that each IB phase transition is finding a component of maximum (nonlinear) correlation between X and Y orthogonal to the learned representation, in close analogy with canonical-correlation analysis (CCA) in linear settings. Based on the theory, we present an algorithm for discovering phase transition points. Finally, we verify that our theory and algorithm accurately predict phase transitions in categorical datasets, predict the onset of learning new classes and class difficulty in MNIST, and predict prominent phase transitions in CIFAR10.