mlsanityData to 7 Oct 2026

Optimal bump functions for shallow ReLU networks: Weight decay, depth separation and the curse of dimensionality

arXiv 2209.01173 ↗First posted 2 Sep 2022stat.ML cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In this note, we study how neural networks with a single hidden layer and ReLU activation interpolate data drawn from a radially symmetric distribution with target labels 1 at the origin and 0 outside the unit ball, if no labels are known inside the unit ball. With weight decay regularization and in the infinite neuron, infinite data limit, we prove that a unique radially symmetric minimizer exists, whose weight decay regularizer and Lipschitz constant grow as d and √d respectively. We furthermore show that the weight decay regularizer grows exponentially in d if the label 1 is imposed on a ball of radius varepsilon rather than just at the origin. By comparison, a neural networks with two hidden layers can approximate the target function without encountering the curse of dimensionality.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 25 of its 51 references that are in the map. First posted on arXiv 2 Sep 2022.

Builds on

25 of 51 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  3. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  4. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  5. 2016Benefits of depth in neural networkscited by 248
  6. 2015The Power of Depth for Feedforward Neural Networkscited by 181
  7. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  8. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 94
  9. 2021Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networkscited by 86
  10. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
  11. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  12. 2016Training Sparse Neural Networkscited by 73
13 more
  1. 2020Representation formulas and pointwise properties for Barron functionscited by 62
  2. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  3. 2019Approximation spaces of deep neural networkscited by 51
  4. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 51
  5. 2019Machine Learning from a Continuous Viewpointcited by 50
  6. 2021Characterization of the Variation Spaces Corresponding to Shallow Neural Networkscited by 48
  7. 2020Neural network approximation and estimation of classifiers with classification boundary in a Barron classcited by 28
  8. 2020Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernelscited by 19
  9. 2020Complexity Measures for Neural Networks with General Activation Functions Using Path-based Normscited by 19
  10. 2019A Priori Estimates of the Population Risk for Residual Networkscited by 14
  11. 2021Ridgeless Interpolation with Shallow ReLU Networks in 1D is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functionscited by 8
  12. 2019Approximation Rates for Neural Networks with General Activation Functionscited by 3
  13. 2021A Note on the Representation Power of GHHscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 2.

4 papers.

  1. 2023Minimum norm interpolation by perceptra: Explicit regularization and implicit biascited by 3
  2. 2024Solving the Poisson Equation with Dirichlet data by shallow ReLUα-networks: A regularity and approximation perspectivecited by 2
  3. 2026Elliptic Regularity Theory in Barron Spaces and Applications to the Deep Ritz Methodcited by 1
  4. 2026The Barron-Lipschitz Energy Gap and Depth Separation Phenomena in Scientific Machine Learningcited by 0