# Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck

arXiv: https://arxiv.org/abs/1712.01272 · HTML: https://mlsanity.com/paper/1712.01272/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2017.

## Built on by (most cited first, 4 of 4)

- [Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression](https://mlsanity.com/paper/2305.08013/) (2023)
- [Information Bottleneck: Exact Analysis of (Quantized) Neural Networks](https://mlsanity.com/paper/2106.12912/) (2021)
- [Lightweight Conceptual Dictionary Learning for Text Classification Using Information Compression](https://mlsanity.com/paper/2405.01584/) (2024)
- [Information Plane Analysis of Binary Neural Networks](https://mlsanity.com/paper/2605.03636/) (2026)

## Abstract (arXiv)

Information Bottleneck (IB) is a generalization of rate-distortion theory that naturally incorporates compression and relevance trade-offs for learning. Though the original IB has been extensively studied, there has not been much understanding of multiple bottlenecks which better fit in the context of neural networks. In this work, we propose Information Multi-Bottlenecks (IMBs) as an extension of IB to multiple bottlenecks which has a direct application to training neural networks by considering layers as multiple bottlenecks and weights as parameterized encoders and decoders. We show that the multiple optimality of IMB is not simultaneously achievable for stochastic encoders. We thus propose a simple compromised scheme of IMB which in turn generalizes maximum likelihood estimate (MLE) principle in the context of stochastic neural networks. We demonstrate the effectiveness of IMB on classification tasks and adversarial robustness in MNIST and CIFAR10.