# Rethinking Neural Networks With Benford's Law

arXiv: https://arxiv.org/abs/2102.03313 · HTML: https://mlsanity.com/paper/2102.03313/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 30 references that are in the map.
- First posted on arXiv 5 Feb 2021.

## Builds on (most cited first, 9 of 9)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [On the use of Benford's law to detect GAN-generated images](https://mlsanity.com/paper/2004.07682/) (2020)
- [TherML: Thermodynamics of Machine Learning](https://mlsanity.com/paper/1807.04162/) (2018)
- [A Free-Energy Principle for Representation Learning](https://mlsanity.com/paper/2002.12406/) (2020)

## Built on by (most cited first, 2 of 2)

- [On the Detection of Anomalous or Out-Of-Distribution Data in Vision Models Using Statistical Techniques](https://mlsanity.com/paper/2403.15497/) (2024)
- [Benford's Law as a Distributional Prior for Post-Training Quantization of Large Language Models](https://mlsanity.com/paper/2602.00165/) (2026)

## Abstract (arXiv)

Benford's Law (BL) or the Significant Digit Law defines the probability distribution of the first digit of numerical values in a data sample. This Law is observed in many naturally occurring datasets. It can be seen as a measure of naturalness of a given distribution and finds its application in areas like anomaly and fraud detection. In this work, we address the following question: Is the distribution of the Neural Network parameters related to the network's generalization capability? To that end, we first define a metric, MLH (Model Enthalpy), that measures the closeness of a set of numbers to Benford's Law and we show empirically that it is a strong predictor of Validation Accuracy. Second, we use MLH as an alternative to Validation Accuracy for Early Stopping, removing the need for a Validation set. We provide experimental evidence that even if the optimal size of the validation set is known before-hand, the peak test accuracy attained is lower than not using a validation set at all. Finally, we investigate the connection of BL to Free Energy Principle and First Law of Thermodynamics, showing that MLH is a component of the internal energy of the learning system and optimization as an analogy to minimizing the total energy to attain equilibrium.