mlsanityData to 14 Oct 2026

On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics

arXiv 2007.15623 ↗First posted 30 Jul 2020stat.ML cs.LG math.FAcited by 39

Not on the map. Why?

Abstract (arXiv)

We develop Banach spaces for ReLU neural networks of finite depth L and infinite width. The spaces contain all finite fully connected L-layer networks and their L2-limiting objects under bounds on the natural path-norm. Under this norm, the unit ball in the space for L-layer networks has low Rademacher complexity and thus favorable generalization properties. Functions in these spaces can be approximated by multi-layer neural networks with dimension-independent convergence rates. The key to this work is a new way of representing functions in some form of expectations, motivated by multi-layer neural networks. This representation allows us to define a new class of continuous models for machine learning. We show that the gradient flow defined this way is the natural continuous analog of the gradient descent dynamics for the associated multi-layer neural networks. We show that the path-norm increases at most polynomially under this continuous gradient flow dynamics.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 8; 2023: 5; 2024: 6; 2025: 4; 2026: 3.

39 papers.

Sort:

By area:

  1. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 96
  2. 2021When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?cited by 58
  3. 2021Can stable and accurate neural networks be computed? -- On the barriers of deep learning and Smale's 18th problemcited by 53
  4. 2024Numerical analysis of physics-informed neural networks and related models in physics-informed machine learningcited by 39
  5. 2021Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionalitycited by 34
  6. 2023Deep Learning Meets Sparse Regularization: A Signal Processing Perspectivecited by 21
  7. 2023A Rainbow in Deep Network Black Boxescited by 18
  8. 2021Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalizationcited by 16
  9. 2022Duality for Neural Networks through Reproducing Kernel Banach Spacescited by 15
  10. 2023A Brief Survey on the Approximation Theory for Sequence Modellingcited by 15
  11. 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 13
  12. 2022Two-Scale Gradient Descent Ascent Dynamics Finds Mixed Nash Equilibria of Continuous Games: A Mean-Field Perspectivecited by 13
27 more
  1. 2024Score-based generative models break the curse of dimensionality in learning a family of sub-Gaussian probability distributionscited by 12
  2. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  3. 2021Optimal learning of high-dimensional classification problems using deep neural networkscited by 9
  4. 2022Provably convergent quasistatic dynamics for mean-field two-player zero-sum gamescited by 9
  5. 2021Generalization Error of GAN from the Discriminator's Perspectivecited by 8
  6. 2021Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networkscited by 8
  7. 2021Training Integrable Parameterizations of Deep Neural Networks in the Infinite-Width Limitcited by 7
  8. 2023Infinite-dimensional reservoir computingcited by 7
  9. 2021A Class of Dimension-free Metrics for the Convergence of Empirical Measurescited by 5
  10. 2022Gradient descent provably escapes saddle points in the training of shallow ReLU networkscited by 5
  11. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  12. 2024ReLUs Are Sufficient for Learning Implicit Neural Representationscited by 5
  13. 2021Solving the Dirichlet problem for the Monge-Amp`ere equation using neural networkscited by 3
  14. 2022A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networkscited by 3
  15. 2022Quantifying the Impact of Label Noise on Federated Learningcited by 3
  16. 2024Nonuniform random feature models using derivative informationcited by 2
  17. 2024Solving the Poisson Equation with Dirichlet data by shallow ReLUα-networks: A regularity and approximation perspectivecited by 2
  18. 2025Nonlocal techniques for the analysis of deep ReLU neural network approximationscited by 2
  19. 2025Convergence of Time-Averaged Mean Field Gradient Descent Dynamics for Continuous Multi-Player Zero-Sum Gamescited by 2
  20. 2025Vector-Valued Reproducing Kernel Banach Spaces for Neural Networks and Operatorscited by 2
  21. 2022Interval Bound Interpolation for Few-shot Learning with Few Taskscited by 1
  22. 2021Depth and Feature Learning are Provably Beneficial for Neural Network Discriminatorscited by 0
  23. 2024A Mean Field Ansatz for Zero-Shot Weight Transfercited by 0
  24. 2025Path Regularization: A Near-Complete and Optimal Nonasymptotic Generalization Theory for Multilayer Neural Networks and Double Descent Phenomenoncited by 0
  25. 2026Featured Reproducing Kernel Banach Spaces for Learning and Neural Networkscited by 0
  26. 2026A First-Principles Theory of Slow Thinking and Active Perceptioncited by 0
  27. 2026Fractional Parabolic Partial Differential Equations in Anisotropic Spectral Barron Spaces: Regularity and Neural Approximationcited by 0