# Caveats for information bottleneck in deterministic scenarios

arXiv: https://arxiv.org/abs/1808.07593 · HTML: https://mlsanity.com/paper/1808.07593/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Aug 2018.

## Built on by (most cited first, 30 of 34)

- [ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction](https://mlsanity.com/paper/2105.10446/) (2021)
- [Examining and Combating Spurious Features under Distribution Shift](https://mlsanity.com/paper/2106.07171/) (2021)
- [Lossy Compression for Lossless Prediction](https://mlsanity.com/paper/2106.10800/) (2021)
- [PAC-Bayes Information Bottleneck](https://mlsanity.com/paper/2109.14509/) (2021)
- [Deep Deterministic Information Bottleneck with Matrix-based Entropy Functional](https://mlsanity.com/paper/2102.00533/) (2021)
- [An Information Criterion for Controlled Disentanglement of Multimodal Data](https://mlsanity.com/paper/2410.23996/) (2024)
- [Bottlenecks CLUB: Unifying Information-Theoretic Trade-offs Among Complexity, Leakage, and Utility](https://mlsanity.com/paper/2207.04895/) (2022)
- [Minimum Description Length and Generalization Guarantees for Representation Learning](https://mlsanity.com/paper/2402.03254/) (2024)
- [Information decomposition in complex systems via machine learning](https://mlsanity.com/paper/2307.04755/) (2023)
- [Cauchy-Schwarz Divergence Information Bottleneck for Regression](https://mlsanity.com/paper/2404.17951/) (2024)
- [Perturbation Theory for the Information Bottleneck](https://mlsanity.com/paper/2105.13977/) (2021)
- [Layerwise Change of Knowledge in Neural Networks](https://mlsanity.com/paper/2409.08712/) (2024)
- [The Causal Information Bottleneck and Optimal Causal Variable Abstractions](https://mlsanity.com/paper/2410.00535/) (2024)
- [Gated Information Bottleneck for Generalization in Sequential Environments](https://mlsanity.com/paper/2110.06057/) (2021)
- [Disentangled Representation Learning with Transmitted Information Bottleneck](https://mlsanity.com/paper/2311.01686/) (2023)
- [Partial information decomposition: redundancy as information bottleneck](https://mlsanity.com/paper/2405.07665/) (2024)
- [Towards Worst-Case Guarantees with Scale-Aware Interpretability](https://mlsanity.com/paper/2602.05184/) (2026)
- [Neuronal Correlation: a Central Concept in Neural Network](https://mlsanity.com/paper/2201.09069/) (2022)
- [Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priors](https://mlsanity.com/paper/2502.15540/) (2025)
- [The SIGReg Objective as Variational Free Energy: A Theoretical Active-Inference Account of JEPA World Models](https://mlsanity.com/paper/2607.13612/) (2026)
- [Dynamic Encoding and Decoding of Information for Split Learning in Mobile-Edge Computing: Leveraging Information Bottleneck Theory](https://mlsanity.com/paper/2309.02787/) (2023)
- [Elastic Information Bottleneck](https://mlsanity.com/paper/2311.03955/) (2023)
- [Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training](https://mlsanity.com/paper/2402.01238/) (2024)
- [Information Plane Analysis Visualization in Deep Learning via Transfer Entropy](https://mlsanity.com/paper/2404.01364/) (2024)
- [Deep Privacy Funnel Model: From a Discriminative to a Generative Approach with an Application to Face Recognition](https://mlsanity.com/paper/2404.02696/) (2024)
- [Why pre-training is beneficial for downstream classification tasks?](https://mlsanity.com/paper/2410.08455/) (2024)
- [Multiview Representation Learning via Distributed Joint Latent Space Structuring](https://mlsanity.com/paper/2504.18455/) (2025)
- [Source-Side Sufficiency for the Information Bottleneck: Exact Reduction and Finite-Block Equivalence](https://mlsanity.com/paper/2604.26744/) (2026)
- [Analysis of Information Flow Through U-Nets](https://mlsanity.com/paper/2101.08427/) (2021)
- [A Reflection on Learning from Data: Epistemology Issues and Limitations](https://mlsanity.com/paper/2107.13270/) (2021)

## Abstract (arXiv)

Information bottleneck (IB) is a method for extracting information from one random variable X that is relevant for predicting another random variable Y. To do so, IB identifies an intermediate "bottleneck" variable T that has low mutual information I(X;T) and high mutual information I(Y;T). The "IB curve" characterizes the set of bottleneck variables that achieve maximal I(Y;T) for a given I(X;T), and is typically explored by maximizing the "IB Lagrangian", I(Y;T) - β I(X;T). In some cases, Y is a deterministic function of X, including many classification problems in supervised learning where the output class Y is a deterministic function of the input X. We demonstrate three caveats when using IB in any situation where Y is a deterministic function of X: (1) the IB curve cannot be recovered by maximizing the IB Lagrangian for different values of β; (2) there are "uninteresting" trivial solutions at all points of the IB curve; and (3) for multi-layer classifiers that achieve low prediction error, different layers cannot exhibit a strict trade-off between compression and prediction, contrary to a recent proposal. We also show that when Y is a small perturbation away from being a deterministic function of X, these three caveats arise in an approximate way. To address problem (1), we propose a functional that, unlike the IB Lagrangian, can recover the IB curve in all cases. We demonstrate the three caveats on the MNIST dataset.