# Information Bottleneck: Exact Analysis of (Quantized) Neural Networks

arXiv: https://arxiv.org/abs/2106.12912 · HTML: https://mlsanity.com/paper/2106.12912/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 18 of 18)

- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization](https://mlsanity.com/paper/1908.09375/) (2019)
- [Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle](https://mlsanity.com/paper/1802.09766/) (2018)
- [Entropy and mutual information in models of deep neural networks](https://mlsanity.com/paper/1805.09785/) (2018)
- [On Information Plane Analyses of Neural Network Classifiers -- A Review](https://mlsanity.com/paper/2003.09671/) (2020)
- [Estimating Mixture Entropy with Pairwise Distances](https://mlsanity.com/paper/1706.02419/) (2017)
- [Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration](https://mlsanity.com/paper/1804.06537/) (2018)
- [Scalable Mutual Information Estimation using Dependence Graphs](https://mlsanity.com/paper/1801.09125/) (2018)
- [Adaptive Estimators Show Information Compression in Deep Neural Networks](https://mlsanity.com/paper/1902.09037/) (2019)
- [Information in Infinite Ensembles of Infinitely-Wide Neural Networks](https://mlsanity.com/paper/1911.09189/) (2019)
- [Information Plane Analysis of Deep Neural Networks via Matrix-Based Renyi's Entropy and Tensor Kernels](https://mlsanity.com/paper/1909.11396/) (2019)
- [What Information Does a ResNet Compress?](https://mlsanity.com/paper/2003.06254/) (2020)
- [Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck](https://mlsanity.com/paper/1712.01272/) (2017)
- [Understanding Learning Dynamics of Binary Neural Networks via Information Bottleneck](https://mlsanity.com/paper/2006.07522/) (2020)

## Built on by (most cited first, 15 of 15)

- [Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression](https://mlsanity.com/paper/2305.08013/) (2023)
- [Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models](https://mlsanity.com/paper/2402.19465/) (2024)
- [Learning in PINNs: Phase transition, diffusion equilibrium, and generalization](https://mlsanity.com/paper/2403.18494/) (2024)
- [Cauchy-Schwarz Divergence Information Bottleneck for Regression](https://mlsanity.com/paper/2404.17951/) (2024)
- [End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training](https://mlsanity.com/paper/2402.09050/) (2024)
- [Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs](https://mlsanity.com/paper/2601.22709/) (2026)
- [InfoQ: Mixed-Precision Quantization via Global Information Flow](https://mlsanity.com/paper/2508.04753/) (2025)
- [Explaining Grokking and Information Bottleneck through Neural Collapse Emergence](https://mlsanity.com/paper/2509.20829/) (2025)
- [Structured IB: Improving Information Bottleneck with Structured Feature Learning](https://mlsanity.com/paper/2412.08222/) (2024)
- [Fisher information flow in artificial neural networks](https://mlsanity.com/paper/2509.02407/) (2025)
- [Understanding and Leveraging the Learning Phases of Neural Networks](https://mlsanity.com/paper/2312.06887/) (2023)
- [The learning phases in NN: From Fitting the Majority to Fitting a Few](https://mlsanity.com/paper/2202.08299/) (2022)
- [Mutual information estimation for graph convolutional neural networks](https://mlsanity.com/paper/2203.16887/) (2022)
- [Information Plane Analysis of Binary Neural Networks](https://mlsanity.com/paper/2605.03636/) (2026)
- [Compression Through Time: An Interplay of Separability, Stationarity and Capacity](https://mlsanity.com/paper/2610.16660/) (2026)

## Abstract (arXiv)

The information bottleneck (IB) principle has been suggested as a way to analyze deep neural networks. The learning dynamics are studied by inspecting the mutual information (MI) between the hidden layers and the input and output. Notably, separate fitting and compression phases during training have been reported. This led to some controversy including claims that the observations are not reproducible and strongly dependent on the type of activation function used as well as on the way the MI is estimated. Our study confirms that different ways of binning when computing the MI lead to qualitatively different results, either supporting or refusing IB conjectures. To resolve the controversy, we study the IB principle in settings where MI is non-trivial and can be computed exactly. We monitor the dynamics of quantized neural networks, that is, we discretize the whole deep learning system so that no approximation is required when computing the MI. This allows us to quantify the information flow without measurement errors. In this setting, we observed a fitting phase for all layers and a compression phase for the output layer in all experiments; the compression in the hidden layers was dependent on the type of activation function. Our study shows that the initial IB results were not artifacts of binning when computing the MI. However, the critical claim that the compression phase may not be observed for some networks also holds true.