# Understanding Neural Networks with Logarithm Determinant Entropy Estimator

arXiv: https://arxiv.org/abs/2105.03705 · HTML: https://mlsanity.com/paper/2105.03705/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 65 references that are in the map.
- First posted on arXiv 8 May 2021.

## Builds on (most cited first, 16 of 16)

- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle](https://mlsanity.com/paper/1802.09766/) (2018)
- [Entropy and mutual information in models of deep neural networks](https://mlsanity.com/paper/1805.09785/) (2018)
- [On Information Plane Analyses of Neural Network Classifiers -- A Review](https://mlsanity.com/paper/2003.09671/) (2020)
- [Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration](https://mlsanity.com/paper/1804.06537/) (2018)
- [Incremental Learning via Rate Reduction](https://mlsanity.com/paper/2011.14593/) (2020)
- [Understanding Autoencoders with Information Theoretic Concepts](https://mlsanity.com/paper/1804.00057/) (2018)
- [Measuring Dependence with Matrix-based Entropy Functional](https://mlsanity.com/paper/2101.10160/) (2021)
- [Deep Deterministic Information Bottleneck with Matrix-based Entropy Functional](https://mlsanity.com/paper/2102.00533/) (2021)
- [Adaptive Estimators Show Information Compression in Deep Neural Networks](https://mlsanity.com/paper/1902.09037/) (2019)
- [On the Information Plane of Autoencoders](https://mlsanity.com/paper/2005.07783/) (2020)
- [Deep Networks from the Principle of Rate Reduction](https://mlsanity.com/paper/2010.14765/) (2020)
- [Variational Predictive Information Bottleneck](https://mlsanity.com/paper/1910.10831/) (2019)
- [Nonparanormal Information Estimation](https://mlsanity.com/paper/1702.07803/) (2017)
- [Do Compressed Representations Generalize Better?](https://mlsanity.com/paper/1909.09706/) (2019)

## Built on by (most cited first, 10 of 10)

- [INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection](https://mlsanity.com/paper/2402.03744/) (2024)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [Does Representation Matter? Exploring Intermediate Layers in Large Language Models](https://mlsanity.com/paper/2412.09563/) (2024)
- [An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization](https://mlsanity.com/paper/2303.00633/) (2023)
- [Understanding Self-Supervised Learning via Latent Distribution Matching](https://mlsanity.com/paper/2605.03517/) (2026)
- [Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry](https://mlsanity.com/paper/2306.04810/) (2023)
- [Improving Semantic Uncertainty Quantification in LVLMs with Semantic Gaussian Processes](https://mlsanity.com/paper/2512.14177/) (2025)
- [Distance Is All You Need: Radial Dispersion for Uncertainty Estimation in Large Language Models](https://mlsanity.com/paper/2512.04351/) (2025)
- [Learning Uncertainty from Sequential Internal Dispersion in Large Language Models](https://mlsanity.com/paper/2604.15741/) (2026)
- [Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation](https://mlsanity.com/paper/2605.27136/) (2026)

## Abstract (arXiv)

Understanding the informative behaviour of deep neural networks is challenged by misused estimators and the complexity of network structure, which leads to inconsistent observations and diversified interpretation. Here we propose the LogDet estimator -- a reliable matrix-based entropy estimator that approximates Shannon differential entropy. We construct informative measurements based on LogDet estimator, verify our method with comparable experiments and utilize it to analyse neural network behaviour. Our results demonstrate the LogDet estimator overcomes the drawbacks that emerge from highly diverse and degenerated distribution thus is reliable to estimate entropy in neural networks. The Network analysis results also find a functional distinction between shallow and deeper layers, which can help understand the compression phenomenon in the Information bottleneck theory of neural networks.