mlsanityData to 14 Oct 2026

Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration

arXiv 1804.06537 ↗First posted 18 Apr 2018cs.LG cs.IT math.ITcited by 32

Not on the map. Why?

Abstract (arXiv)

The matrix-based Renyi's α-entropy functional and its multivariate extension were recently developed in terms of the normalized eigenspectrum of a Hermitian matrix of the projected data in a reproducing kernel Hilbert space (RKHS). However, the utility and possible applications of these new estimators are rather new and mostly unknown to practitioners. In this paper, we first show that our estimators enable straightforward measurement of information flow in realistic convolutional neural networks (CNN) without any approximation. Then, we introduce the partial information decomposition (PID) framework and develop three quantities to analyze the synergy and redundancy in convolutional layer representations. Our results validate two fundamental data processing inequalities and reveal some fundamental properties concerning the training of CNN.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 5; 2023: 6; 2024: 3; 2025: 2; 2026: 5.

32 papers.

Sort:

By area:

  1. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 115
  2. 2024Knowledge Distillation Based on Transformed Teacher Matchingcited by 37
  3. 2021Measuring Dependence with Matrix-based Entropy Functionalcited by 19
  4. 2021Information Theoretic Representation Distillationcited by 19
  5. 2023Information Bottleneck Analysis of Deep Neural Networks via Lossy Compressioncited by 19
  6. 2021Deep Deterministic Information Bottleneck with Matrix-based Entropy Functionalcited by 18
  7. 2021Information Bottleneck: Exact Analysis of (Quantized) Neural Networkscited by 15
  8. 2022A Measure of the Complexity of Neural Representations based on Partial Information Decompositioncited by 11
  9. 2021Understanding Neural Networks with Logarithm Determinant Entropy Estimatorcited by 10
  10. 2021Disentanglement Analysis with Partial Information Decompositioncited by 9
  11. 2024Cauchy-Schwarz Divergence Information Bottleneck for Regressioncited by 9
  12. 2022Synergistic information supports modality integration and flexible learning in neural networks solving multiple taskscited by 8
20 more
  1. 2022HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labelscited by 7
  2. 2021Rethinking Hard-Parameter Sharing in Multi-Domain Learningcited by 6
  3. 2023New Adversarial Image Detection Based on Sentiment Analysiscited by 6
  4. 2026A Brain-like Synergistic Core in LLMs Drives Behaviour and Learningcited by 6
  5. 2021Computationally Efficient Approximations for Matrix-based Renyi's Entropycited by 4
  6. 2023Training Invertible Neural Networks as Autoencoderscited by 4
  7. 2025Optimal Transport Depth Up-Scalingcited by 4
  8. 2024What should a neuron aim for? Designing local objective functions based on information theorycited by 3
  9. 2023Filter Pruning based on Information Capacity and Independencecited by 2
  10. 2023Wavelet Dynamic Selection Network for Inertial Sensor Signal Enhancementcited by 2
  11. 2026PIME: Prototype-based Interpretable MCTS-Enhanced Brain Network Analysis for Disorder Diagnosiscited by 2
  12. 2022Higher-order mutual information reveals synergistic sub-networks for multi-neuron importancecited by 1
  13. 2021The distance between the weights of the neural network is meaningfulcited by 0
  14. 2021A Reflection on Learning from Data: Epistemology Issues and Limitationscited by 0
  15. 2022Mutual information estimation for graph convolutional neural networkscited by 0
  16. 2023MOLE: MOdular Learning FramEwork via Mutual Information Maximizationcited by 0
  17. 2025Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognitioncited by 0
  18. 2026Anatomy of a failure: When, how, and why deep vision fails in scientific domainscited by 0
  19. 2026Seeing Before Answering: Training-Free Visual Layer Profiling for Vision-Language Modelscited by 0
  20. 2026Layer-wise Input Recoverability in Deep Neural Networkscited by 0