mlsanityData to 14 Oct 2026

Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Space

arXiv 2307.01177 ↗First posted 3 Jul 2023cs.LG math.FA math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

To characterize the function space explored by neural networks (NNs) is an important aspect of learning theory. In this work, noticing that a multi-layer NN generates implicitly a hierarchy of reproducing kernel Hilbert spaces (RKHSs) - named a neural Hilbert ladder (NHL) - we define the function space as an infinite union of RKHSs, which generalizes the existing Barron space theory of two-layer NNs. We then establish several theoretical properties of the new space. First, we prove a correspondence between functions expressed by L-layer NNs and those belonging to L-level NHLs. Second, we prove generalization guarantees for learning an NHL with a controlled complexity measure. Third, we derive a non-Markovian dynamics of random fields that governs the evolution of the NHL which is induced by the training of multi-layer NNs in an infinite-width mean-field limit. Fourth, we show examples of depth separation in NHLs under the ReLU activation function. Finally, we perform numerical experiments to illustrate the feature learning aspect of NN training through the lens of NHLs.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 101 of its 148 references that are in the map. First posted on arXiv 3 Jul 2023.

Builds on

101 of 148 references.

References · 101

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  5. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  6. 2018On Lazy Training in Differentiable Programmingcited by 675
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  8. 2017Deep Neural Networks as Gaussian Processescited by 622
  9. 2016Error bounds for approximations with deep ReLU networkscited by 574
  10. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  11. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  12. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
38 more
  1. 2015Deep Kernel Learningcited by 445
  2. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  3. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  4. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  5. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  6. 2017Implicit Regularization in Matrix Factorizationcited by 322
  7. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  8. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  9. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  10. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  11. 2016Benefits of depth in neural networkscited by 250
  12. 2016Deep Learning without Poor Local Minimacited by 234
  13. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  14. 2015The Power of Depth for Feedforward Neural Networkscited by 183
  15. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  16. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  17. 2021The Principles of Deep Learning Theorycited by 173
  18. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  19. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  20. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  21. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  22. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  23. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  24. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 138
  25. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  26. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
  27. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  28. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  29. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  30. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  31. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 114
  32. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 106
  33. 2019How do infinite width bounded norm networks look in function space?cited by 106
  34. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  35. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  36. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  37. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 85
  38. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85

50 most cited of 101 All 101 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 2.

5 papers.

  1. 2025Deep Learning as a Convex Paradigm of Computation: Minimizing Circuit Size with ResNetscited by 5
  2. 2024Solving the Poisson Equation with Dirichlet data by shallow ReLUα-networks: A regularity and approximation perspectivecited by 2
  3. 2023Learning a Sparse Representation of Barron Functions with the Inverse Scale Space Flowcited by 1
  4. 2026Sobolev Approximation of Deep ReLU Networks in Log-Barron Spacecited by 1
  5. 2026A Deep Risk Estimator for Known Operator Learningcited by 1