# The deterministic information bottleneck

arXiv: https://arxiv.org/abs/1604.00268 · HTML: https://mlsanity.com/paper/1604.00268/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2016.

## Built on by (most cited first, 30 of 42)

- [Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.06607/) (2021)
- [From Canonical Correlation Analysis to Self-supervised Graph Neural Networks](https://mlsanity.com/paper/2106.12484/) (2021)
- [Deep Deterministic Information Bottleneck with Matrix-based Entropy Functional](https://mlsanity.com/paper/2102.00533/) (2021)
- [Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Exploration](https://mlsanity.com/paper/2103.12300/) (2021)
- [Task-Oriented Communication for Edge Video Analytics](https://mlsanity.com/paper/2211.14049/) (2022)
- [Bottlenecks CLUB: Unifying Information-Theoretic Trade-offs Among Complexity, Leakage, and Utility](https://mlsanity.com/paper/2207.04895/) (2022)
- [Cauchy-Schwarz Divergence Information Bottleneck for Regression](https://mlsanity.com/paper/2404.17951/) (2024)
- [MvHo-IB: Multi-View Higher-Order Information Bottleneck for Brain Disorder Diagnosis](https://mlsanity.com/paper/2507.02847/) (2025)
- [Optimal Randomized Approximations for Matrix based Renyi's Entropy](https://mlsanity.com/paper/2205.07426/) (2022)
- [Data efficiency, dimensionality reduction, and the generalized symmetric information bottleneck](https://mlsanity.com/paper/2309.05649/) (2023)
- [Bounding generalization error with input compression: An empirical study with infinite-width networks](https://mlsanity.com/paper/2207.09408/) (2022)
- [Coping with Change: Learning Invariant and Minimum Sufficient Representations for Fine-Grained Visual Categorization](https://mlsanity.com/paper/2306.04893/) (2023)
- [The Causal Information Bottleneck and Optimal Causal Variable Abstractions](https://mlsanity.com/paper/2410.00535/) (2024)
- [Gated Information Bottleneck for Generalization in Sequential Environments](https://mlsanity.com/paper/2110.06057/) (2021)
- [Conditional entropy minimization principle for learning domain invariant representation features](https://mlsanity.com/paper/2201.10460/) (2022)
- [An Information Bottleneck Characterization of the Understanding-Workload Tradeoff](https://mlsanity.com/paper/2310.07802/) (2023)
- [Modeling Higher-Order Brain Interactions via a Multi-View Information Bottleneck Framework for fMRI-based Psychiatric Diagnosis](https://mlsanity.com/paper/2604.17713/) (2026)
- [Statistically Valid Information Bottleneck via Multiple Hypothesis Testing](https://mlsanity.com/paper/2409.07325/) (2024)
- [Studying the Interplay between Information Loss and Operation Loss in Representations for Classification](https://mlsanity.com/paper/2112.15238/) (2021)
- [Classification Utility, Fairness, and Compactness via Tunable Information Bottleneck and R'enyi Measures](https://mlsanity.com/paper/2206.10043/) (2022)
- [A Linear Programming Approach for Resource-Aware Information-Theoretic Tree Abstractions](https://mlsanity.com/paper/2208.04220/) (2022)
- [Elastic Information Bottleneck](https://mlsanity.com/paper/2311.03955/) (2023)
- [Privacy for Fairness: Information Obfuscation for Fair Representation Learning with Local Differential Privacy](https://mlsanity.com/paper/2402.10473/) (2024)
- [Deep Privacy Funnel Model: From a Discriminative to a Generative Approach with an Application to Face Recognition](https://mlsanity.com/paper/2404.02696/) (2024)
- [Understanding Encoder-Decoder Structures in Machine Learning Using Information Measures](https://mlsanity.com/paper/2405.20452/) (2024)
- [A Deterministic Information Bottleneck Method for Clustering Mixed-Type Data](https://mlsanity.com/paper/2407.03389/) (2024)
- [Topic Identification in LLM Input-Output Pairs through the Lens of Information Bottleneck](https://mlsanity.com/paper/2509.03533/) (2025)
- [The Information-Theoretic Imperative: Compression and the Epistemic Foundations of Intelligence](https://mlsanity.com/paper/2510.25883/) (2025)
- [CEPAE: Conditional Entropy-Penalized Autoencoders for Time Series Counterfactuals](https://mlsanity.com/paper/2602.15546/) (2026)
- [Progressive Interpretation Synthesis: Interpreting Task Solving by Quantifying Previously Used and Unused Information](https://mlsanity.com/paper/2101.02879/) (2021)

## Abstract (arXiv)

Lossy compression and clustering fundamentally involve a decision about what features are relevant and which are not. The information bottleneck method (IB) by Tishby, Pereira, and Bialek formalized this notion as an information-theoretic optimization problem and proposed an optimal tradeoff between throwing away as many bits as possible, and selectively keeping those that are most important. In the IB, compression is measure my mutual information. Here, we introduce an alternative formulation that replaces mutual information with entropy, which we call the deterministic information bottleneck (DIB), that we argue better captures this notion of compression. As suggested by its name, the solution to the DIB problem turns out to be a deterministic encoder, or hard clustering, as opposed to the stochastic encoder, or soft clustering, that is optimal under the IB. We compare the IB and DIB on synthetic data, showing that the IB and DIB perform similarly in terms of the IB cost function, but that the DIB significantly outperforms the IB in terms of the DIB cost function. We also empirically find that the DIB offers a considerable gain in computational efficiency over the IB, over a range of convergence parameters. Our derivation of the DIB also suggests a method for continuously interpolating between the soft clustering of the IB and the hard clustering of the DIB.