mlsanityData to 7 Oct 2026

A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case

arXiv 1910.01635 ↗First posted 3 Oct 2019cs.LG stat.MLcited by 113

Not on the map. Why?

Abstract (arXiv)

A key element of understanding the efficacy of overparameterized neural networks is characterizing how they represent functions as the number of weights in the network approaches infinity. In this paper, we characterize the norm required to realize a function f:ℝdrightarrowℝ as a single hidden-layer ReLU network with an unbounded number of units (infinite width), but where the Euclidean norm of the weights is bounded, including precisely characterizing which functions can be realized with finite norm. This was settled for univariate univariate functions in Savarese et al. (2019), where it was shown that the required norm is determined by the L1-norm of the second derivative of the function. We extend the characterization to multivariate functions (i.e., networks with d input units), relating the required norm to the L1-norm of the Radon transform of a (d+1)/2-power Laplacian of the function. This characterization allows us to show that all functions in Sobolev spaces Ws,1(ℝ), s≥ d+1, can be represented with bounded norm, to calculate the required norm for several specific functions, and to obtain a depth separation result. These results have important implications for understanding generalization performance and the distinction between neural networks and more traditional kernel learning.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 22; 2023: 26; 2024: 19; 2025: 9; 2026: 16.

113 papers.

Sort:

By area:

  1. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  2. 2021Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networkscited by 86
  3. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  4. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  5. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  6. 2021Neural Fields as Learnable Kernels for 3D Reconstructioncited by 58
  7. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  8. 2021Understanding neural networks with reproducing kernel Banach spacescited by 50
  9. 2021Characterization of the Variation Spaces Corresponding to Shallow Neural Networkscited by 48
  10. 2021Near-Minimax Optimal Estimation With Shallow ReLU Neural Networkscited by 40
  11. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  12. 2022Flat minima generalize for low-rank matrix recoverycited by 28
38 more
  1. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  2. 2023Nonparametric regression using over-parameterized shallow ReLU neural networkscited by 24
  3. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  4. 2023Deep Learning Meets Sparse Regularization: A Signal Processing Perspectivecited by 21
  5. 2021Depth separation beyond radial functionscited by 18
  6. 2021Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Normcited by 18
  7. 2022Ridges, Neural Networks, and the Radon Transformcited by 18
  8. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  9. 2022Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsitycited by 17
  10. 2023Penalising the biases in norm regularisation enforces sparsitycited by 17
  11. 2023Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compressioncited by 17
  12. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  13. 2022The Role of Linear Layers in Nonlinear Interpolating Networkscited by 15
  14. 2023Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regressioncited by 15
  15. 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15
  16. 2025On the Interpolation Effect of Score Smoothing in Diffusion Modelscited by 15
  17. 2022Fully-Connected Network on Noncompact Symmetric Space and Ridgelet Transform based on Helgason-Fourier Analysiscited by 14
  18. 2021Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networkscited by 13
  19. 2022Active Learning with Neural Networks: Insights from Nonparametric Statisticscited by 13
  20. 2022Knowledge Distillation: Bad Models Can Be Good Role Modelscited by 12
  21. 2022On the Robustness of Bayesian Neural Networks to Adversarial Attackscited by 12
  22. 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
  23. 2022Universality of Group Convolutional Neural Networks Based on Ridgelet Analysis on Groupscited by 11
  24. 2023Optimal Approximation of Zonoids and Uniform Approximation by Shallow Neural Networkscited by 11
  25. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  26. 2022Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?cited by 10
  27. 2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reusecited by 9
  28. 2023How do Minimum-Norm Shallow Denoisers Look in Function Space?cited by 9
  29. 2021On Energy-Based Models with Overparametrized Shallow Neural Networkscited by 8
  30. 2021Ridgeless Interpolation with Shallow ReLU Networks in 1D is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functionscited by 8
  31. 2024Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networkscited by 8
  32. 2021Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Spacecited by 7
  33. 2021A spectral-based analysis of the separation between two-layer neural networks and linear methodscited by 7
  34. 2023ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Modelscited by 7
  35. 2023Weighted variation spaces and approximation by shallow ReLU networkscited by 7
  36. 2023Extending Path-Dependent NJ-ODEs to Noisy Observations and a Dependent Observation Frameworkcited by 6
  37. 2024Approximation Rates for Shallow ReLUk Neural Networks on Sobolev Spaces via the Radon Transformcited by 6
  38. 2025When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Netscited by 6

50 most cited of 113 All 113 (JSON)