# Bayesian Free Energy of Deep ReLU Neural Network in Overparametrized Cases

arXiv: https://arxiv.org/abs/2303.15739 · HTML: https://mlsanity.com/paper/2303.15739/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 26 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Builds on (most cited first, 3 of 3)

- [Deep Learning is Singular, and That's Good](https://mlsanity.com/paper/2010.11560/) (2020)
- [A Bayesian information criterion for singular models](https://mlsanity.com/paper/1309.0911/) (2013)
- [The Exact Asymptotic Form of Bayesian Generalization Error in Latent Dirichlet Allocation](https://mlsanity.com/paper/2008.01304/) (2020)

## Built on by (most cited first, 5 of 5)

- [Free energy of Bayesian Convolutional Neural Network with Skip Connection](https://mlsanity.com/paper/2307.01417/) (2023)
- [SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training](https://mlsanity.com/paper/2505.23489/) (2025)
- [Upper Bound of Bayesian Generalization Error in Partial Concept Bottleneck Model (CBM): Partial CBM outperforms naive CBM](https://mlsanity.com/paper/2403.09206/) (2024)
- [Statistical inference for quantum singular models](https://mlsanity.com/paper/2411.16396/) (2024)
- [PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory](https://mlsanity.com/paper/2604.17219/) (2026)

## Abstract (arXiv)

In many research fields in artificial intelligence, it has been shown that deep neural networks are useful to estimate unknown functions on high dimensional input spaces. However, their generalization performance is not yet completely clarified from the theoretical point of view because they are nonidentifiable and singular learning machines. Moreover, a ReLU function is not differentiable, to which algebraic or analytic methods in singular learning theory cannot be applied. In this paper, we study a deep ReLU neural network in overparametrized cases and prove that the Bayesian free energy, which is equal to the minus log marginal likelihoodor the Bayesian stochastic complexity, is bounded even if the number of layers are larger than necessary to estimate an unknown data-generating function. Since the Bayesian generalization error is equal to the increase of the free energy as a function of a sample size, our result also shows that the Bayesian generalization error does not increase even if a deep ReLU neural network is designed to be sufficiently large or in an opeverparametrized state.