# Upper Bounds for Local Learning Coefficients of Three-Layer Neural Networks

arXiv: https://arxiv.org/abs/2603.12785 · HTML: https://mlsanity.com/paper/2603.12785/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 49 references that are in the map.
- First posted on arXiv 13 Mar 2026.

## Builds on (most cited first, 7 of 7)

- [The Local Learning Coefficient: A Singularity-Aware Complexity Measure](https://mlsanity.com/paper/2308.12108/) (2023)
- [Loss Landscape Degeneracy and Stagewise Development in Transformers](https://mlsanity.com/paper/2402.02364/) (2024)
- [Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient](https://mlsanity.com/paper/2410.02984/) (2024)
- [A Bayesian information criterion for singular models](https://mlsanity.com/paper/1309.0911/) (2013)
- [Marginal likelihood and model selection for Gaussian latent tree and forest models](https://mlsanity.com/paper/1412.8285/) (2014)
- [Estimating Real Log Canonical Thresholds](https://mlsanity.com/paper/1906.01341/) (2019)
- [Asymptotic Behavior of Bayesian Generalization Error in Multinomial Mixtures](https://mlsanity.com/paper/2203.06884/) (2022)

## Built on by (most cited first, 1 of 1)

- [A Statistical Difference between Single-Layer Learning and Hierarchical Learning in Wide Neural Networks](https://mlsanity.com/paper/2607.23397/) (2026)

## Abstract (arXiv)

Three-layer neural networks are known to form singular learning models, and their Bayesian asymptotic behavior is governed by the learning coefficient, or real log canonical threshold. Although this quantity has been clarified for regular models and for some special singular models, broadly applicable methods for evaluating it in neural networks remain limited. Recently, a formula for the local learning coefficient of semiregular models was proposed, yielding an upper bound on the learning coefficient. However, this formula applies only to nonsingular points in the set of realization parameters and cannot be used at singular points. In particular, for three-layer neural networks, the resulting upper bound has been shown to differ substantially from learning coefficient values already known in some cases. In this paper, we derive a formula for an upper bound on local learning coefficients at a class of singular realization parameters in three-layer neural networks. This formula can be interpreted as a counting rule under budget, demand, and supply constraints. In the non-polynomial real-analytic case, the formula applies in general settings, whereas in the polynomial case it applies under the restriction that the true distribution has no hidden units. In particular, our result covers activation functions such as the swish function and also includes polynomial activation functions under the above restriction, thereby extending previous results to a broader class of activation functions. We further show that, when the input dimension is one, the numerical value given by the right-hand side of our upper-bound formula agrees with the previously known learning coefficient, thereby providing a useful comparison with known exact results. Our result also provides a systematic perspective on how the weight parameters of three-layer neural networks affect the learning coefficient.