mlsanityData to 14 Oct 2026

Spectral complexity of deep neural networks

arXiv 2405.09541 ↗First posted 15 May 2024stat.ML cs.LG math.PRcited by 3

Not on the map. Why?

Abstract (arXiv)

It is well-known that randomly initialized, push-forward, fully-connected neural networks weakly converge to isotropic Gaussian processes, in the limit where the width of all layers goes to infinity. In this paper, we propose to use the angular power spectrum of the limiting field to characterize the complexity of the network architecture. In particular, we define sequences of random variables associated with the angular power spectrum, and provide a full characterization of the network complexity in terms of the asymptotic distribution of these sequences as the depth diverges. On this basis, we classify neural networks as low-disorder, sparse, or high-disorder; we show how this classification highlights a number of distinct features for standard activation functions, and in particular, sparsity properties of ReLU networks. Our theoretical results are also validated by numerical simulations.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 35 of its 65 references that are in the map. First posted on arXiv 15 May 2024.

Builds on

35 of 65 references.

References · 35

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  3. 2017Deep Neural Networks as Gaussian Processescited by 622
  4. 2015Deep Kernel Learningcited by 445
  5. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  6. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  7. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  8. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 311
  9. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  10. 2016Benefits of depth in neural networkscited by 250
  11. 2016Deep Information Propagationcited by 195
  12. 2015The Power of Depth for Feedforward Neural Networkscited by 183
23 more
  1. 2019Complexity of Linear Regions in Deep Networkscited by 167
  2. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 161
  3. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  4. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 102
  5. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 85
  6. 2016Deep vs. shallow networks : An approximation theory perspectivecited by 63
  7. 2016Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networkscited by 59
  8. 2017Depth Separation for Neural Networkscited by 46
  9. 2019Nonlinear Approximation and (Deep) ReLU Networkscited by 45
  10. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
  11. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  12. 2017A representer theorem for deep kernel learningcited by 26
  13. 2022Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networkscited by 25
  14. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  15. 2023Quantitative CLTs in Deep Neural Networkscited by 20
  16. 2021Depth separation beyond radial functionscited by 18
  17. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  18. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  19. 2021Rate of Convergence of Polynomial Networks to Gaussian Processescited by 13
  20. 2023Gaussian random field approximation via Stein's method with applications to wide random neural networkscited by 11
  21. 2023A Quantitative Functional Central Limit Theorem for Shallow Neural Networkscited by 8
  22. 2021Measuring Complexity of Learning Schemes Using Hessian-Schatten Total Variationcited by 4
  23. 2024Depth Separation in Norm-Bounded Infinite-Width Neural Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Fractal and Regular Geometry of Deep Neural Networkscited by 1
  2. 2025Critical Points of Random Neural Networkscited by 0
  3. 2026A Rigorous, Tractable Measure of Model Complexitycited by 0