# Perturbation Theory for the Information Bottleneck

arXiv: https://arxiv.org/abs/2105.13977 · HTML: https://mlsanity.com/paper/2105.13977/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 39 references that are in the map.
- First posted on arXiv 28 May 2021.

## Builds on (most cited first, 12 of 12)

- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Emergence of Invariance and Disentanglement in Deep Representations](https://mlsanity.com/paper/1706.01350/) (2017)
- [Information Dropout: Learning Optimal Representations Through Noisy Computation](https://mlsanity.com/paper/1611.01353/) (2016)
- [InfoBot: Transfer and Exploration via the Information Bottleneck](https://mlsanity.com/paper/1901.10902/) (2019)
- [Caveats for information bottleneck in deterministic scenarios](https://mlsanity.com/paper/1808.07593/) (2018)
- [Learning Optimal Representations with the Decodable Information Bottleneck](https://mlsanity.com/paper/2009.12789/) (2020)
- [Learnability for the Information Bottleneck](https://mlsanity.com/paper/1907.07331/) (2019)
- [Predictive information in a sensory population](https://mlsanity.com/paper/1307.0225/) (2013)
- [Phase Transitions for the Information Bottleneck in Representation Learning](https://mlsanity.com/paper/2001.01878/) (2020)
- [The information bottleneck and geometric clustering](https://mlsanity.com/paper/1712.09657/) (2017)
- [Discovering Potential Correlations via Hypercontractivity](https://mlsanity.com/paper/1709.04024/) (2017)

## Built on by (most cited first, 6 of 6)

- [Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality](https://mlsanity.com/paper/2208.03848/) (2022)
- [Computationally Efficient Approximations for Matrix-based Renyi's Entropy](https://mlsanity.com/paper/2112.13720/) (2021)
- [Generalized Information Bottleneck for Gaussian Variables](https://mlsanity.com/paper/2303.17762/) (2023)
- [A representation-learning game for classes of prediction tasks](https://mlsanity.com/paper/2403.06971/) (2024)
- [Pareto-optimal clustering with the primal deterministic information bottleneck](https://mlsanity.com/paper/2204.02489/) (2022)
- [The Information Bottleneck's Ordinary Differential Equation: First-Order Root-Tracking for the IB](https://mlsanity.com/paper/2306.09790/) (2023)

## Abstract (arXiv)

Extracting relevant information from data is crucial for all forms of learning. The information bottleneck (IB) method formalizes this, offering a mathematically precise and conceptually appealing framework for understanding learning phenomena. However the nonlinearity of the IB problem makes it computationally expensive and analytically intractable in general. Here we derive a perturbation theory for the IB method and report the first complete characterization of the learning onset, the limit of maximum relevant information per bit extracted from data. We test our results on synthetic probability distributions, finding good agreement with the exact numerical solution near the onset of learning. We explore the difference and subtleties in our derivation and previous attempts at deriving a perturbation theory for the learning onset and attribute the discrepancy to a flawed assumption. Our work also provides a fresh perspective on the intimate relationship between the IB method and the strong data processing inequality.