mlsanityData to 14 Oct 2026

Minimum norm interpolation by perceptra: Explicit regularization and implicit bias

arXiv 2311.06138 ↗First posted 10 Nov 2023stat.ML cs.LG math.OCcited by 3

Not on the map. Why?

Abstract (arXiv)

We investigate how shallow ReLU networks interpolate between known regions. Our analysis shows that empirical risk minimizers converge to a minimum norm interpolant as the number of data points and parameters tends to infinity when a weight decay regularizer is penalized with a coefficient which vanishes at a precise rate as the network width and the number of data points grow. With and without explicit regularization, we numerically study the implicit bias of common optimization algorithms towards known minimum norm interpolants.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 46 of its 82 references that are in the map. First posted on arXiv 10 Nov 2023.

Builds on

46 of 82 references.

References · 46

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2018On Lazy Training in Differentiable Programmingcited by 675
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  7. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  8. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  9. 2019Two models of double descent for weak featurescited by 195
  10. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  11. 2020Implicit Gradient Regularizationcited by 146
  12. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
34 more
  1. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  2. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 114
  3. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  4. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  5. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  6. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  7. 2019How do infinite width bounded norm networks look in function space?cited by 106
  8. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  9. 2021Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networkscited by 86
  10. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  11. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  12. 2020Representation formulas and pointwise properties for Barron functionscited by 62
  13. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  14. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 52
  15. 2019Machine Learning from a Continuous Viewpointcited by 50
  16. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  17. 2021Characterization of the Variation Spaces Corresponding to Shallow Neural Networkscited by 48
  18. 2020A Brief Prehistory of Double Descentcited by 44
  19. 2018The loss landscape of overparameterized neural networkscited by 39
  20. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 39
  21. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  22. 2020Neural network approximation and estimation of classifiers with classification boundary in a Barron classcited by 28
  23. 2018Spurious Valleys in Two-layer Neural Network Optimization Landscapescited by 21
  24. 2020Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernelscited by 19
  25. 2020Complexity Measures for Neural Networks with General Activation Functions Using Path-based Normscited by 19
  26. 2023Penalising the biases in norm regularisation enforces sparsitycited by 19
  27. 2022The Role of Linear Layers in Nonlinear Interpolating Networkscited by 16
  28. 2020Implicit Bias of Gradient Descent for Mean Squared Error Regression with Two-Layer Wide Neural Networkscited by 15
  29. 2020Sparsest Piecewise-Linear Regression of One-Dimensional Datacited by 12
  30. 2021Ridgeless Interpolation with Shallow ReLU Networks in 1D is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functionscited by 9
  31. 2022On the symmetries in the dynamics of wide two-layer neural networkscited by 8
  32. 2022Optimal bump functions for shallow ReLU networks: Weight decay, depth separation and the curse of dimensionalitycited by 4
  33. 2019Scaling Limit of Neural Networks with the Xavier Initialization and Convergence to a Global Minimumcited by 3
  34. 2023On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Elliptic Regularity Theory in Barron Spaces and Applications to the Deep Ritz Methodcited by 1
  2. 2026The Barron-Lipschitz Energy Gap and Depth Separation Phenomena in Scientific Machine Learningcited by 0
  3. 2026The double descent and Runge phenomena in overparametrized polynomial interpolationcited by 0