# Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration

arXiv: https://arxiv.org/abs/1804.06537 · HTML: https://mlsanity.com/paper/1804.06537/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Apr 2018.

## Built on by (most cited first, 30 of 32)

- [Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges](https://mlsanity.com/paper/2103.04505/) (2021)
- [Knowledge Distillation Based on Transformed Teacher Matching](https://mlsanity.com/paper/2402.11148/) (2024)
- [Measuring Dependence with Matrix-based Entropy Functional](https://mlsanity.com/paper/2101.10160/) (2021)
- [Information Theoretic Representation Distillation](https://mlsanity.com/paper/2112.00459/) (2021)
- [Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression](https://mlsanity.com/paper/2305.08013/) (2023)
- [Deep Deterministic Information Bottleneck with Matrix-based Entropy Functional](https://mlsanity.com/paper/2102.00533/) (2021)
- [Information Bottleneck: Exact Analysis of (Quantized) Neural Networks](https://mlsanity.com/paper/2106.12912/) (2021)
- [A Measure of the Complexity of Neural Representations based on Partial Information Decomposition](https://mlsanity.com/paper/2209.10438/) (2022)
- [Understanding Neural Networks with Logarithm Determinant Entropy Estimator](https://mlsanity.com/paper/2105.03705/) (2021)
- [Disentanglement Analysis with Partial Information Decomposition](https://mlsanity.com/paper/2108.13753/) (2021)
- [Cauchy-Schwarz Divergence Information Bottleneck for Regression](https://mlsanity.com/paper/2404.17951/) (2024)
- [Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks](https://mlsanity.com/paper/2210.02996/) (2022)
- [HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels](https://mlsanity.com/paper/2202.10716/) (2022)
- [Rethinking Hard-Parameter Sharing in Multi-Domain Learning](https://mlsanity.com/paper/2107.11359/) (2021)
- [New Adversarial Image Detection Based on Sentiment Analysis](https://mlsanity.com/paper/2305.03173/) (2023)
- [A Brain-like Synergistic Core in LLMs Drives Behaviour and Learning](https://mlsanity.com/paper/2601.06851/) (2026)
- [Computationally Efficient Approximations for Matrix-based Renyi's Entropy](https://mlsanity.com/paper/2112.13720/) (2021)
- [Training Invertible Neural Networks as Autoencoders](https://mlsanity.com/paper/2303.11239/) (2023)
- [Optimal Transport Depth Up-Scaling](https://mlsanity.com/paper/2508.08011/) (2025)
- [What should a neuron aim for? Designing local objective functions based on information theory](https://mlsanity.com/paper/2412.02482/) (2024)
- [Filter Pruning based on Information Capacity and Independence](https://mlsanity.com/paper/2303.03645/) (2023)
- [Wavelet Dynamic Selection Network for Inertial Sensor Signal Enhancement](https://mlsanity.com/paper/2401.05416/) (2023)
- [PIME: Prototype-based Interpretable MCTS-Enhanced Brain Network Analysis for Disorder Diagnosis](https://mlsanity.com/paper/2602.21046/) (2026)
- [Higher-order mutual information reveals synergistic sub-networks for multi-neuron importance](https://mlsanity.com/paper/2211.00416/) (2022)
- [The distance between the weights of the neural network is meaningful](https://mlsanity.com/paper/2102.00396/) (2021)
- [A Reflection on Learning from Data: Epistemology Issues and Limitations](https://mlsanity.com/paper/2107.13270/) (2021)
- [Mutual information estimation for graph convolutional neural networks](https://mlsanity.com/paper/2203.16887/) (2022)
- [MOLE: MOdular Learning FramEwork via Mutual Information Maximization](https://mlsanity.com/paper/2308.07772/) (2023)
- [Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognition](https://mlsanity.com/paper/2510.27651/) (2025)
- [Anatomy of a failure: When, how, and why deep vision fails in scientific domains](https://mlsanity.com/paper/2605.04231/) (2026)

## Abstract (arXiv)

The matrix-based Renyi's α-entropy functional and its multivariate extension were recently developed in terms of the normalized eigenspectrum of a Hermitian matrix of the projected data in a reproducing kernel Hilbert space (RKHS). However, the utility and possible applications of these new estimators are rather new and mostly unknown to practitioners. In this paper, we first show that our estimators enable straightforward measurement of information flow in realistic convolutional neural networks (CNN) without any approximation. Then, we introduce the partial information decomposition (PID) framework and develop three quantities to analyze the synergy and redundancy in convolutional layer representations. Our results validate two fundamental data processing inequalities and reveal some fundamental properties concerning the training of CNN.