# Exact capacity of the wide hidden layer treelike neural networks with generic activations

arXiv: https://arxiv.org/abs/2402.05719 · HTML: https://mlsanity.com/paper/2402.05719/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2024.

## Built on by (most cited first, 6 of 6)

- [Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networks](https://mlsanity.com/paper/2410.06717/) (2024)
- [Injectivity capacity of ReLU gates](https://mlsanity.com/paper/2410.20646/) (2024)
- [Solution space and storage capacity of fully connected two-layer neural networks with generic activation functions](https://mlsanity.com/paper/2404.13404/) (2024)
- [Deep ReLU networks -- injectivity capacity upper bounds](https://mlsanity.com/paper/2412.19677/) (2024)
- [Capacity of the Hebbian-Hopfield network associative memory](https://mlsanity.com/paper/2403.01907/) (2024)
- [Exact objectives of random linear programs and mean widths of random polyhedrons](https://mlsanity.com/paper/2403.03637/) (2024)

## Abstract (arXiv)

Recent progress in studying treelike committee machines (TCM) neural networks (NN) in citeStojnictcmspnncaprdt23,Stojnictcmspnncapliftedrdt23,Stojnictcmspnncapdiffactrdt23 showed that the Random Duality Theory (RDT) and its a partially lifted(pl RDT) variant are powerful tools that can be used for very precise networks capacity analysis. Here, we consider wide hidden layer networks and uncover that certain aspects of numerical difficulties faced in citeStojnictcmspnncapdiffactrdt23 miraculously disappear. In particular, we employ recently developed fully lifted (fl) RDT to characterize the wide (drightarrow ∞) TCM nets capacity. We obtain explicit, closed form, capacity characterizations for a very generic class of the hidden layer activations. While the utilized approach significantly lowers the amount of the needed numerical evaluations, the ultimate fl RDT usefulness and success still require a solid portion of the residual numerical work. To get the concrete capacity values, we take four very famous activations examples: emphReLU, textbfquadratic, textbferf, and textbftanh. After successfully conducting all the residual numerical work for all of them, we uncover that the whole lifting mechanism exhibits a remarkably rapid convergence with the relative improvements no better than ~ 0.1% happening already on the 3-rd level of lifting. As a convenient bonus, we also uncover that the capacity characterizations obtained on the first and second level of lifting precisely match those obtained through the statistical physics replica theory methods in citeZavPeh21 for the generic and in citeBalMalZech19 for the ReLU activations.