mlsanityData to 14 Oct 2026

Entropy and mutual information in models of deep neural networks

arXiv 1805.09785 ↗First posted 24 May 2018cs.LG cond-mat.dis-nn cs.ITcited by 51

Not on the map. Why?

Abstract (arXiv)

We examine a class of deep learning models with a tractable method to compute information-theoretic quantities. Our contributions are three-fold: (i) We show how entropies and mutual informations can be derived from heuristic statistical physics methods, under the assumption that weight matrices are independent and orthogonally-invariant. (ii) We extend particular cases in which this result is known to be rigorously exact by providing a proof for two-layers networks with Gaussian random weights, using the recently introduced adaptive interpolation method. (iii) We propose an experiment framework with generative models of synthetic datasets, on which we train deep neural networks with a weight constraint designed so that the assumption in (i) is verified during learning. We study the behavior of entropies and mutual informations throughout learning and conclude that, in the proposed setting, the relationship between compression and generalization remains elusive.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 10; 2023: 10; 2024: 8; 2025: 12; 2026: 4.

51 papers.

Sort:

By area:

  1. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  2. 2023GiGaMAE: Generalizable Graph Masked Autoencoder via Collaborative Latent Space Reconstructioncited by 21
  3. 2023Information Bottleneck Analysis of Deep Neural Networks via Lossy Compressioncited by 19
  4. 2022On double-descent in uncertainty quantification in overparametrized modelscited by 16
  5. 2021Information Bottleneck: Exact Analysis of (Quantized) Neural Networkscited by 15
  6. 2021Towards quantifying information flows: relative entropy in deep neural networks and the renormalization groupcited by 14
  7. 2021On the interplay between data structure and loss function in classification problemscited by 13
  8. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 13
  9. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  10. 2021Deep learning via message passing algorithms based on belief propagationcited by 12
  11. 2024High-dimensional learning of narrow neural networkscited by 12
  12. 2021Understanding Neural Networks with Logarithm Determinant Entropy Estimatorcited by 10
38 more
  1. 2025Learned Hallucination Detection in Black-Box LLMs using Token-level Entropy Production Ratecited by 10
  2. 2022The price of ignorance: how much does it cost to forget noise structure in low-rank matrix estimation?cited by 9
  3. 2022On the generalization of learning algorithms that do not convergecited by 9
  4. 2022Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensionscited by 8
  5. 2024Progress Measures for Grokking on Real-world Taskscited by 8
  6. 2023High-Dimensional Smoothed Entropy Estimation via Dimensionality Reductioncited by 7
  7. 2022Monitoring Shortcut Learning using Mutual Informationcited by 6
  8. 2023Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamicscited by 6
  9. 2025Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regimecited by 6
  10. 2022Bounding generalization error with input compression: An empirical study with infinite-width networkscited by 4
  11. 2023Neural-prior stochastic block modelcited by 4
  12. 2024Exploring Loss Landscapes through the Lens of Spin Glass Theorycited by 4
  13. 2025High-entropy Advantage in Neural Networks' Generalizabilitycited by 4
  14. 2021On the training of sparse and dense deep neural networks: less parameters, same performancecited by 3
  15. 2022Bayes-optimal limits in structured PCA, and how to reach themcited by 3
  16. 2022Approximate Message Passing for Multi-Layer Estimation in Rotationally Invariant Modelscited by 3
  17. 2023Entropy-based Guidance of Deep Neural Networks for Accelerated Convergence and Improved Performancecited by 3
  18. 2024Fed-pilot: Optimizing LoRA Allocation for Efficient Federated Fine-Tuning with Heterogeneous Clientscited by 3
  19. 2021Review-Based Domain Disentanglement without Duplicate Users or Contexts for Cross-Domain Recommendationcited by 2
  20. 2025Estimating Time Series Foundation Model Transferability via In-Context Learningcited by 2
  21. 2025TELL-TALE: Task Efficient LLMs with Task Aware Layer Eliminationcited by 2
  22. 2025Dropouts in Confidence: Moral Uncertainty in Human-LLM Alignmentcited by 2
  23. 2026Why Do Unlearnable Examples Work: A Novel Perspective of Mutual Informationcited by 2
  24. 2022Deep radiomic signature with immune cell markers predicts the survival of glioma patientscited by 1
  25. 2024Information Plane Analysis Visualization in Deep Learning via Transfer Entropycited by 1
  26. 2024Lightweight Conceptual Dictionary Learning for Text Classification Using Information Compressioncited by 1
  27. 2025FALCON: Fine-grained Activation Manipulation by Contrastive Orthogonal Unalignment for Large Language Modelcited by 1
  28. 2025ACMamba: Fast Unsupervised Anomaly Detection via An Asymmetrical Consensus State Space Modelcited by 1
  29. 2025Rethinking the Understanding Ability across LLMs through Mutual Informationcited by 1
  30. 2025A Spin Glass Characterization of Neural Networkscited by 1
  31. 2025IBNorm: Information-Bottleneck Inspired Normalization for Representation Learningcited by 1
  32. 2022k-Sliced Mutual Information: A Quantitative Study of Scalability with Dimensioncited by 0
  33. 2023Spatially heterogeneous learning by a deep student machinecited by 0
  34. 2023An information-Theoretic Approach to Semi-supervised Transfer Learningcited by 0
  35. 2024Informative Subgraphs Aware Masked Auto-Encoder in Dynamic Graphscited by 0
  36. 2025Knowledge-Informed Neural Network for Complex-Valued SAR Image Recognitioncited by 0
  37. 2026TAPIOCA: Why Task- Aware Pruning Improves OOD model Capabilitycited by 0
  38. 2026Random Label Prediction Heads for Studying Memorization in Deep Neural Networkscited by 0

50 most cited of 51 All 51 (JSON)