mlsanityData to 7 Oct 2026

Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression

arXiv 2305.08013 ↗First posted 13 May 2023cs.LG cs.IT math.ITcited by 19

Not on the map. Why?

Abstract (arXiv)

The Information Bottleneck (IB) principle offers an information-theoretic framework for analyzing the training process of deep neural networks (DNNs). Its essence lies in tracking the dynamics of two mutual information (MI) values: between the hidden layer output and the DNN input/target. According to the hypothesis put forth by Shwartz-Ziv & Tishby (2017), the training process consists of two distinct phases: fitting and compression. The latter phase is believed to account for the good generalization performance exhibited by DNNs. Due to the challenging nature of estimating MI between high-dimensional random vectors, this hypothesis was only partially verified for NNs of tiny sizes or specific types, such as quantized NNs. In this paper, we introduce a framework for conducting IB analysis of general NNs. Our approach leverages the stochastic NN method proposed by Goldfeld et al. (2019) and incorporates a compression step to overcome the obstacles associated with high dimensionality. In other words, we estimate the MI between the compressed representations of high-dimensional random vectors. The proposed method is supported by both theoretical and practical justifications. Notably, we demonstrate the accuracy of our estimator through synthetic experiments featuring predefined MI values and comparison with MINE (Belghazi et al., 2018). Finally, we perform IB analysis on a close-to-real-scale convolutional DNN, which reveals new features of the MI dynamics.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 32 of its 72 references that are in the map. First posted on arXiv 13 May 2023.

Builds on

32 of 72 references.

References · 32

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2015Variational Inference with Normalizing Flowscited by 2,161
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  5. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  6. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  7. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  8. 2018MINE: Mutual Information Neural Estimationcited by 680
  9. 2019On Variational Bounds of Mutual Informationcited by 550
  10. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  11. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  12. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
20 more
  1. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  2. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 145
  3. 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
  4. 2018Entropy and mutual information in models of deep neural networkscited by 50
  5. 2017Model-Powered Conditional Independence Testcited by 43
  6. 2020On Information Plane Analyses of Neural Network Classifiers -- A Reviewcited by 35
  7. 2018Understanding Convolutional Neural Networks with Information Theory: An Initial Explorationcited by 31
  8. 2016Efficient multivariate entropy estimation via k-nearest neighbour distancescited by 26
  9. 2017Nonparametric independence testing via mutual informationcited by 22
  10. 2018Scalable Mutual Information Estimation using Dependence Graphscited by 22
  11. 2020Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learningcited by 16
  12. 2017Optimal rates of entropy estimation over Lipschitz ballscited by 15
  13. 2021Information Bottleneck: Exact Analysis of (Quantized) Neural Networkscited by 14
  14. 2018Understanding Neural Networks and Individual Neuron Importance via Information-Ordered Cumulative Ablationcited by 13
  15. 2019Information Plane Analysis of Deep Neural Networks via Matrix-Based Renyi's Entropy and Tensor Kernelscited by 11
  16. 2023High-Dimensional Smoothed Entropy Estimation via Dimensionality Reductioncited by 7
  17. 2019Constrained deep neural network architecture search for IoT devices accounting hardware calibrationcited by 6
  18. 2020What Information Does a ResNet Compress?cited by 5
  19. 2017Layer-wise Learning of Stochastic Neural Networks with Information Bottleneckcited by 4
  20. 2022Invariant Representations with Stochastically Quantized Neural Networkscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 10; 2026: 6.

19 papers.

Sort:

By area:

  1. 2024Mutual Information Estimation via Normalizing Flowscited by 22
  2. 2025InfoBridge: Mutual Information estimation via Bridge Matchingcited by 7
  3. 2026Gated Relational Alignment via Confidence-based Distillation for Efficient VLMscited by 5
  4. 2024Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMaxcited by 4
  5. 2025InfoQ: Mixed-Precision Quantization via Global Information Flowcited by 4
  6. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  7. 2025Neural Mutual Information Estimation with Vector Copulascited by 4
  8. 2025FMMI: Flow Matching Mutual Information Estimationcited by 4
  9. 2025Bottlenecked Transformers: Periodic KV Cache Consolidation for Generalised Reasoningcited by 3
  10. 2025Binarized Neural Networks Converge Toward Algorithmic Simplicity: Empirical Support for the Learning-as-Compression Hypothesiscited by 3
  11. 2025Information-Bottleneck Driven Binary Neural Network for Change Detectioncited by 2
  12. 2025Curse of Slicing: Why Sliced Mutual Information is a Deceptive Measure of Statistical Dependencecited by 1
7 more
  1. 2025Multimodal Datasets with Controllable Mutual Informationcited by 1
  2. 2026Discrete Bridges for Mutual Information Estimationcited by 1
  3. 2026InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimatecited by 1
  4. 2024Real Time American Sign Language Detection Using Yolo-v9cited by 0
  5. 2026GO-GenZip: Goal-Oriented Generative Sampling and Hybrid Compressioncited by 0
  6. 2026Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimationcited by 0
  7. 2026Training-Adaptive Convolutional Sparse Coding via Information Bottleneck for Robust Visual Representationcited by 0