# Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression

arXiv: https://arxiv.org/abs/2305.08013 · HTML: https://mlsanity.com/paper/2305.08013/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 72 references that are in the map.
- First posted on arXiv 13 May 2023.

## Builds on (most cited first, 30 of 32)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [Information-theoretic analysis of generalization capability of learning algorithms](https://mlsanity.com/paper/1705.07809/) (2017)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Reasoning About Generalization via Conditional Mutual Information](https://mlsanity.com/paper/2001.09122/) (2020)
- [Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle](https://mlsanity.com/paper/1802.09766/) (2018)
- [Entropy and mutual information in models of deep neural networks](https://mlsanity.com/paper/1805.09785/) (2018)
- [Model-Powered Conditional Independence Test](https://mlsanity.com/paper/1709.06138/) (2017)
- [On Information Plane Analyses of Neural Network Classifiers -- A Review](https://mlsanity.com/paper/2003.09671/) (2020)
- [Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration](https://mlsanity.com/paper/1804.06537/) (2018)
- [Efficient multivariate entropy estimation via k-nearest neighbour distances](https://mlsanity.com/paper/1606.00304/) (2016)
- [Nonparametric independence testing via mutual information](https://mlsanity.com/paper/1711.06642/) (2017)
- [Scalable Mutual Information Estimation using Dependence Graphs](https://mlsanity.com/paper/1801.09125/) (2018)
- [Optimal rates of entropy estimation over Lipschitz balls](https://mlsanity.com/paper/1711.02141/) (2017)
- [Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning](https://mlsanity.com/paper/2003.12537/) (2020)
- [Information Bottleneck: Exact Analysis of (Quantized) Neural Networks](https://mlsanity.com/paper/2106.12912/) (2021)
- [Understanding Neural Networks and Individual Neuron Importance via Information-Ordered Cumulative Ablation](https://mlsanity.com/paper/1804.06679/) (2018)
- [Information Plane Analysis of Deep Neural Networks via Matrix-Based Renyi's Entropy and Tensor Kernels](https://mlsanity.com/paper/1909.11396/) (2019)
- [High-Dimensional Smoothed Entropy Estimation via Dimensionality Reduction](https://mlsanity.com/paper/2305.04712/) (2023)
- [Constrained deep neural network architecture search for IoT devices accounting hardware calibration](https://mlsanity.com/paper/1909.10818/) (2019)
- [What Information Does a ResNet Compress?](https://mlsanity.com/paper/2003.06254/) (2020)

## Built on by (most cited first, 19 of 19)

- [Mutual Information Estimation via Normalizing Flows](https://mlsanity.com/paper/2403.02187/) (2024)
- [InfoBridge: Mutual Information estimation via Bridge Matching](https://mlsanity.com/paper/2502.01383/) (2025)
- [Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs](https://mlsanity.com/paper/2601.22709/) (2026)
- [Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMax](https://mlsanity.com/paper/2410.06993/) (2024)
- [InfoQ: Mixed-Precision Quantization via Global Information Flow](https://mlsanity.com/paper/2508.04753/) (2025)
- [Explaining Grokking and Information Bottleneck through Neural Collapse Emergence](https://mlsanity.com/paper/2509.20829/) (2025)
- [Neural Mutual Information Estimation with Vector Copulas](https://mlsanity.com/paper/2510.20968/) (2025)
- [FMMI: Flow Matching Mutual Information Estimation](https://mlsanity.com/paper/2511.08552/) (2025)
- [Bottlenecked Transformers: Periodic KV Cache Consolidation for Generalised Reasoning](https://mlsanity.com/paper/2505.16950/) (2025)
- [Binarized Neural Networks Converge Toward Algorithmic Simplicity: Empirical Support for the Learning-as-Compression Hypothesis](https://mlsanity.com/paper/2505.20646/) (2025)
- [Information-Bottleneck Driven Binary Neural Network for Change Detection](https://mlsanity.com/paper/2507.03504/) (2025)
- [Curse of Slicing: Why Sliced Mutual Information is a Deceptive Measure of Statistical Dependence](https://mlsanity.com/paper/2506.04053/) (2025)
- [Multimodal Datasets with Controllable Mutual Information](https://mlsanity.com/paper/2510.21686/) (2025)
- [Discrete Bridges for Mutual Information Estimation](https://mlsanity.com/paper/2602.08894/) (2026)
- [InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate](https://mlsanity.com/paper/2606.00241/) (2026)
- [Real Time American Sign Language Detection Using Yolo-v9](https://mlsanity.com/paper/2407.17950/) (2024)
- [GO-GenZip: Goal-Oriented Generative Sampling and Hybrid Compression](https://mlsanity.com/paper/2603.20109/) (2026)
- [Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation](https://mlsanity.com/paper/2607.03487/) (2026)
- [Training-Adaptive Convolutional Sparse Coding via Information Bottleneck for Robust Visual Representation](https://mlsanity.com/paper/2609.19122/) (2026)

## Abstract (arXiv)

The Information Bottleneck (IB) principle offers an information-theoretic framework for analyzing the training process of deep neural networks (DNNs). Its essence lies in tracking the dynamics of two mutual information (MI) values: between the hidden layer output and the DNN input/target. According to the hypothesis put forth by Shwartz-Ziv & Tishby (2017), the training process consists of two distinct phases: fitting and compression. The latter phase is believed to account for the good generalization performance exhibited by DNNs. Due to the challenging nature of estimating MI between high-dimensional random vectors, this hypothesis was only partially verified for NNs of tiny sizes or specific types, such as quantized NNs. In this paper, we introduce a framework for conducting IB analysis of general NNs. Our approach leverages the stochastic NN method proposed by Goldfeld et al. (2019) and incorporates a compression step to overcome the obstacles associated with high dimensionality. In other words, we estimate the MI between the compressed representations of high-dimensional random vectors. The proposed method is supported by both theoretical and practical justifications. Notably, we demonstrate the accuracy of our estimator through synthetic experiments featuring predefined MI values and comparison with MINE (Belghazi et al., 2018). Finally, we perform IB analysis on a close-to-real-scale convolutional DNN, which reveals new features of the MI dynamics.