mlsanityData to 7 Oct 2026

Understanding Neural Networks with Logarithm Determinant Entropy Estimator

arXiv 2105.03705 ↗First posted 8 May 2021cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

Understanding the informative behaviour of deep neural networks is challenged by misused estimators and the complexity of network structure, which leads to inconsistent observations and diversified interpretation. Here we propose the LogDet estimator -- a reliable matrix-based entropy estimator that approximates Shannon differential entropy. We construct informative measurements based on LogDet estimator, verify our method with comparable experiments and utilize it to analyse neural network behaviour. Our results demonstrate the LogDet estimator overcomes the drawbacks that emerge from highly diverse and degenerated distribution thus is reliable to estimate entropy in neural networks. The Network analysis results also find a functional distinction between shallow and deeper layers, which can help understand the compression phenomenon in the Information bottleneck theory of neural networks.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 16 of its 65 references that are in the map. First posted on arXiv 8 May 2021.

Builds on

16 of 65 references.

References · 16

  1. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  2. 2020Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reductioncited by 141
  3. 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
  4. 2018Entropy and mutual information in models of deep neural networkscited by 50
  5. 2020On Information Plane Analyses of Neural Network Classifiers -- A Reviewcited by 35
  6. 2018Understanding Convolutional Neural Networks with Information Theory: An Initial Explorationcited by 31
  7. 2020Incremental Learning via Rate Reductioncited by 31
  8. 2018Understanding Autoencoders with Information Theoretic Conceptscited by 29
  9. 2021Measuring Dependence with Matrix-based Entropy Functionalcited by 19
  10. 2021Deep Deterministic Information Bottleneck with Matrix-based Entropy Functionalcited by 18
  11. 2019Adaptive Estimators Show Information Compression in Deep Neural Networkscited by 14
  12. 2020On the Information Plane of Autoencoderscited by 10
4 more
  1. 2020Deep Networks from the Principle of Rate Reductioncited by 10
  2. 2019Variational Predictive Information Bottleneckcited by 8
  3. 2017Nonparanormal Information Estimationcited by 4
  4. 2019Do Compressed Representations Generalize Better?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 3; 2026: 3.

10 papers.

  1. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  2. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  3. 2024Does Representation Matter? Exploring Intermediate Layers in Large Language Modelscited by 31
  4. 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
  5. 2026Understanding Self-Supervised Learning via Latent Distribution Matchingcited by 4
  6. 2023Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetrycited by 3
  7. 2025Improving Semantic Uncertainty Quantification in LVLMs with Semantic Gaussian Processescited by 3
  8. 2025Distance Is All You Need: Radial Dispersion for Uncertainty Estimation in Large Language Modelscited by 1
  9. 2026Learning Uncertainty from Sequential Internal Dispersion in Large Language Modelscited by 1
  10. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generationcited by 0