mlsanityData to 14 Oct 2026

Critical Points of Random Neural Networks

arXiv 2505.17000 ↗First posted 22 May 2025stat.ML cs.LG math.PRcited by 0

Not on the map. Why?

Abstract (arXiv)

This work investigates the expected number of critical points of random neural networks with different activation functions as the depth increases in the infinite-width limit. Under suitable regularity conditions, we derive precise asymptotic formulas for the expected number of critical points of fixed index and those exceeding a given threshold. Our analysis reveals three distinct regimes depending on the value of the first derivative of the covariance evaluated at 1: the expected number of critical points may converge, grow polynomially, or grow exponentially with depth. The theoretical predictions are supported by numerical experiments. Moreover, we provide numerical evidence suggesting that, when the regularity condition is not satisfied (e.g. for neural networks with ReLU as activation function), the number of critical points increases as the map resolution increases, indicating a potential divergence in the number of critical points.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 41 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

19 of 41 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Deep Neural Networks as Gaussian Processescited by 622
  3. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  5. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  6. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  7. 2021Deep learning: a statistical viewpointcited by 240
  8. 2019Complexity of Linear Regions in Deep Networkscited by 167
  9. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  10. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 85
  11. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
  12. 2022Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networkscited by 25
7 more
  1. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  2. 2023Quantitative CLTs in Deep Neural Networkscited by 20
  3. 2023Gaussian random field approximation via Stein's method with applications to wide random neural networkscited by 11
  4. 2023A Quantitative Functional Central Limit Theorem for Shallow Neural Networkscited by 8
  5. 2021Measuring Complexity of Learning Schemes Using Hessian-Schatten Total Variationcited by 4
  6. 2024Spectral complexity of deep neural networkscited by 3
  7. 2025Fractal and Regular Geometry of Deep Neural Networkscited by 1

Built on by

Not cited yet.