# On Information Plane Analyses of Neural Network Classifiers -- A Review

arXiv: https://arxiv.org/abs/2003.09671 · HTML: https://mlsanity.com/paper/2003.09671/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Mar 2020.

## Built on by (most cited first, 30 of 37)

- [To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review](https://mlsanity.com/paper/2304.09355/) (2023)
- [Examining and Combating Spurious Features under Distribution Shift](https://mlsanity.com/paper/2106.07171/) (2021)
- [Bridging Information-Theoretic and Geometric Compression in Language Models](https://mlsanity.com/paper/2310.13620/) (2023)
- [Mutual Information Estimation via Normalizing Flows](https://mlsanity.com/paper/2403.02187/) (2024)
- [Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression](https://mlsanity.com/paper/2305.08013/) (2023)
- [Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models](https://mlsanity.com/paper/2402.19465/) (2024)
- [Information Bottleneck: Exact Analysis of (Quantized) Neural Networks](https://mlsanity.com/paper/2106.12912/) (2021)
- [Learning in PINNs: Phase transition, diffusion equilibrium, and generalization](https://mlsanity.com/paper/2403.18494/) (2024)
- [A Measure of the Complexity of Neural Representations based on Partial Information Decomposition](https://mlsanity.com/paper/2209.10438/) (2022)
- [Minimum Description Length and Generalization Guarantees for Representation Learning](https://mlsanity.com/paper/2402.03254/) (2024)
- [Free Lunch in Pathology Foundation Model: Task-specific Model Adaptation with Concept-Guided Feature Enhancement](https://mlsanity.com/paper/2411.09894/) (2024)
- [Understanding Neural Networks with Logarithm Determinant Entropy Estimator](https://mlsanity.com/paper/2105.03705/) (2021)
- [End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training](https://mlsanity.com/paper/2402.09050/) (2024)
- [Explaining Classifiers by Constructing Familiar Concepts](https://mlsanity.com/paper/2203.04109/) (2022)
- [On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learning](https://mlsanity.com/paper/2211.09817/) (2022)
- [Information flows of diverse autoencoders](https://mlsanity.com/paper/2102.07402/) (2021)
- [Adaptation of Autoencoder for Sparsity Reduction From Clinical Notes Representation Learning](https://mlsanity.com/paper/2209.12831/) (2022)
- [InfoQ: Mixed-Precision Quantization via Global Information Flow](https://mlsanity.com/paper/2508.04753/) (2025)
- [Explaining Grokking and Information Bottleneck through Neural Collapse Emergence](https://mlsanity.com/paper/2509.20829/) (2025)
- [A Generalized Information Bottleneck Theory of Deep Learning](https://mlsanity.com/paper/2509.26327/) (2025)
- [Fisher information flow in artificial neural networks](https://mlsanity.com/paper/2509.02407/) (2025)
- [Understanding and Leveraging the Learning Phases of Neural Networks](https://mlsanity.com/paper/2312.06887/) (2023)
- [Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priors](https://mlsanity.com/paper/2502.15540/) (2025)
- [Bayesian Learning for Disparity Map Refinement for Semi-Dense Active Stereo Vision](https://mlsanity.com/paper/2209.05082/) (2022)
- [Information Plane Analysis Visualization in Deep Learning via Transfer Entropy](https://mlsanity.com/paper/2404.01364/) (2024)
- [Understanding the role of autoencoders for stiff dynamical systems using information theory](https://mlsanity.com/paper/2503.06325/) (2025)
- [Multiview Representation Learning via Distributed Joint Latent Space Structuring](https://mlsanity.com/paper/2504.18455/) (2025)
- [The learning phases in NN: From Fitting the Majority to Fitting a Few](https://mlsanity.com/paper/2202.08299/) (2022)
- [Mutual information estimation for graph convolutional neural networks](https://mlsanity.com/paper/2203.16887/) (2022)
- [Exploring Adversarial Examples and Adversarial Robustness of Convolutional Neural Networks by Mutual Information](https://mlsanity.com/paper/2207.05756/) (2022)

## Abstract (arXiv)

We review the current literature concerned with information plane analyses of neural network classifiers. While the underlying information bottleneck theory and the claim that information-theoretic compression is causally linked to generalization are plausible, empirical evidence was found to be both supporting and conflicting. We review this evidence together with a detailed analysis of how the respective information quantities were estimated. Our survey suggests that compression visualized in information planes is not necessarily information-theoretic, but is rather often compatible with geometric compression of the latent representations. This insight gives the information plane a renewed justification. Aside from this, we shed light on the problem of estimating mutual information in deterministic neural networks and its consequences. Specifically, we argue that even in feed-forward neural networks the data processing inequality need not hold for estimates of mutual information. Similarly, while a fitting phase, in which the mutual information between the latent representation and the target increases, is necessary (but not sufficient) for good classification performance, depending on the specifics of mutual information estimation such a fitting phase need not be visible in the information plane.