# Learnability for the Information Bottleneck

arXiv: https://arxiv.org/abs/1907.07331 · HTML: https://mlsanity.com/paper/1907.07331/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2019.

## Built on by (most cited first, 24 of 24)

- [Representation Noising: A Defence Mechanism Against Harmful Finetuning](https://mlsanity.com/paper/2405.14577/) (2024)
- [Lossy Compression for Lossless Prediction](https://mlsanity.com/paper/2106.10800/) (2021)
- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [Bottlenecks CLUB: Unifying Information-Theoretic Trade-offs Among Complexity, Leakage, and Utility](https://mlsanity.com/paper/2207.04895/) (2022)
- [Interpretability with full complexity by constraining feature information](https://mlsanity.com/paper/2211.17264/) (2022)
- [Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation](https://mlsanity.com/paper/2607.21973/) (2026)
- [Perturbation Theory for the Information Bottleneck](https://mlsanity.com/paper/2105.13977/) (2021)
- [Adversarial Information Bottleneck](https://mlsanity.com/paper/2103.00381/) (2021)
- [Adaptive Information Bottleneck Guided Joint Source and Channel Coding for Image Transmission](https://mlsanity.com/paper/2203.06492/) (2022)
- [Transfer Entropy Bottleneck: Learning Sequence to Sequence Information Transfer](https://mlsanity.com/paper/2211.16607/) (2022)
- [A Provably Convergent Information Bottleneck Solution via ADMM](https://mlsanity.com/paper/2102.04729/) (2021)
- [Generalized Information Bottleneck for Gaussian Variables](https://mlsanity.com/paper/2303.17762/) (2023)
- [Geometric Analysis of Reasoning Trajectories: A Phase Space Approach to Understanding Valid and Invalid Multi-Hop Reasoning in LLMs](https://mlsanity.com/paper/2410.04415/) (2024)
- [Pre-train and Fine-tune: Recommenders as Large Models](https://mlsanity.com/paper/2501.14268/) (2025)
- [Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training](https://mlsanity.com/paper/2402.01238/) (2024)
- [Deep Privacy Funnel Model: From a Discriminative to a Generative Approach with an Application to Face Recognition](https://mlsanity.com/paper/2404.02696/) (2024)
- [Pareto-optimal clustering with the primal deterministic information bottleneck](https://mlsanity.com/paper/2204.02489/) (2022)
- [Do Neural Networks Generalize from Self-Averaging Sub-classifiers in the Same Way As Adaptive Boosting?](https://mlsanity.com/paper/2302.06923/) (2023)
- [The Information Bottleneck's Ordinary Differential Equation: First-Order Root-Tracking for the IB](https://mlsanity.com/paper/2306.09790/) (2023)
- [Phase Transitions between Accuracy Regimes in L2 regularized Deep Neural Networks](https://mlsanity.com/paper/2505.06597/) (2025)
- [Information flow in multilayer perceptrons: an in-depth analysis](https://mlsanity.com/paper/2510.13846/) (2025)
- [GeoIB: Geometry-Aware Information Bottleneck via Statistical-Manifold Compression](https://mlsanity.com/paper/2602.03906/) (2026)
- [Information Bottleneck Meets Quantization: Finite Rate Analysis and Optimal Designs](https://mlsanity.com/paper/2606.10869/) (2026)
- [Information Bottleneck under Perfect Privacy](https://mlsanity.com/paper/2608.11003/) (2026)

## Abstract (arXiv)

The Information Bottleneck (IB) method (citetishby2000information) provides an insightful and principled approach for balancing compression and prediction for representation learning. The IB objective I(X;Z)-β I(Y;Z) employs a Lagrange multiplier β to tune this trade-off. However, in practice, not only is β chosen empirically without theoretical guidance, there is also a lack of theoretical understanding between β, learnability, the intrinsic nature of the dataset and model capacity. In this paper, we show that if β is improperly chosen, learning cannot happen -- the trivial representation P(Z|X)=P(Z) becomes the global minimum of the IB objective. We show how this can be avoided, by identifying a sharp phase transition between the unlearnable and the learnable which arises as β is varied. This phase transition defines the concept of IB-Learnability. We prove several sufficient conditions for IB-Learnability, which provides theoretical guidance for choosing a good β. We further show that IB-learnability is determined by the largest confident, typical, and imbalanced subset of the examples (the conspicuous subset), and discuss its relation with model capacity. We give practical algorithms to estimate the minimum β for a given dataset. We also empirically demonstrate our theoretical conditions with analyses of synthetic datasets, MNIST, and CIFAR10.