# On the Difference Between the Information Bottleneck and the Deep Information Bottleneck

arXiv: https://arxiv.org/abs/1912.13480 · HTML: https://mlsanity.com/paper/1912.13480/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2019.

## Built on by (most cited first, 5 of 5)

- [Gated Information Bottleneck for Generalization in Sequential Environments](https://mlsanity.com/paper/2110.06057/) (2021)
- [On Learning Prediction-Focused Mixtures](https://mlsanity.com/paper/2110.13221/) (2021)
- [Learning Conditional Invariance through Cycle Consistency](https://mlsanity.com/paper/2111.13185/) (2021)
- [Information Plane Analysis Visualization in Deep Learning via Transfer Entropy](https://mlsanity.com/paper/2404.01364/) (2024)
- [Tighter Bounds on the Information Bottleneck with Application to Deep Learning](https://mlsanity.com/paper/2402.07639/) (2024)

## Abstract (arXiv)

Combining the Information Bottleneck model with deep learning by replacing mutual information terms with deep neural nets has proved successful in areas ranging from generative modelling to interpreting deep neural networks. In this paper, we revisit the Deep Variational Information Bottleneck and the assumptions needed for its derivation. The two assumed properties of the data X, Y and their latent representation T take the form of two Markov chains T-X-Y and X-T-Y. Requiring both to hold during the optimisation process can be limiting for the set of potential joint distributions P(X,Y,T). We therefore show how to circumvent this limitation by optimising a lower bound for I(T;Y) for which only the latter Markov chain has to be satisfied. The actual mutual information consists of the lower bound which is optimised in DVIB and cognate models in practice and of two terms measuring how much the former requirement T-X-Y is violated. Finally, we propose to interpret the family of information bottleneck models as directed graphical models and show that in this framework the original and deep information bottlenecks are special cases of a fundamental IB model.