mlsanityData to 7 Oct 2026

Noisy Interpolation Learning with Shallow Univariate ReLU Networks

arXiv 2307.15396 ↗First posted 28 Jul 2023cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Understanding how overparameterized neural networks generalize despite perfect interpolation of noisy training data is a fundamental question. Mallinar et. al. 2022 noted that neural networks seem to often exhibit ``tempered overfitting'', wherein the population risk does not converge to the Bayes optimal error, but neither does it approach infinity, yielding non-trivial generalization. However, this has not been studied rigorously. We provide the first rigorous analysis of the overfitting behavior of regression with minimum norm (ℓ2 of weights), focusing on univariate two-layer ReLU networks. We show overfitting is tempered (with high probability) when measured with respect to the L1 loss, but also show that the situation is more complex than suggested by Mallinar et. al., and overfitting is catastrophic with respect to the L2 loss, or when taking an expectation over the training set.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 45 of its 56 references that are in the map. First posted on arXiv 28 Jul 2023.

Builds on

45 of 56 references.

References · 45

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2019Benign Overfitting in Linear Regressioncited by 544
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  4. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  5. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  6. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  7. 2019Two models of double descent for weak featurescited by 194
  8. 2020Benign overfitting in ridge regressioncited by 152
  9. 2019Harmless interpolation of noisy data in regressioncited by 121
  10. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  11. 2019How do infinite width bounded norm networks look in function space?cited by 105
  12. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
33 more
  1. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  2. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  3. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  4. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  5. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  6. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  7. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  8. 2020Convex Geometry and Duality of Over-parameterized Neural Networkscited by 49
  9. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  10. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  11. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  12. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 42
  13. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 41
  14. 2022The Implicit Bias of Benign Overfittingcited by 38
  15. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  16. 2020Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic Viewcited by 32
  17. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  18. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
  19. 2020Binary Classification of Gaussian Mixtures: Abundance of Support Vectors, Benign Overfitting and Regularizationcited by 27
  20. 2020Failures of model-dependent generalization bounds for least-norm interpolationcited by 26
  21. 2021Interpolating Classifiers Make Few Mistakescited by 22
  22. 2022Fast Rates for Noisy Interpolation Require Rethinking the Effects of Inductive Biascited by 21
  23. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  24. 2021The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networkscited by 18
  25. 2023Penalising the biases in norm regularisation enforces sparsitycited by 17
  26. 2021Tight bounds for minimum l1-norm interpolation of noisy datacited by 16
  27. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  28. 2021Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networkscited by 13
  29. 2020Sparsest Piecewise-Linear Regression of One-Dimensional Datacited by 12
  30. 2022A Non-Asymptotic Moreau Envelope Theory for High-Dimensional Generalized Linear Modelscited by 11
  31. 2021Ridgeless Interpolation with Shallow ReLU Networks in 1D is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functionscited by 8
  32. 2022A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictorscited by 7
  33. 2023Interpolation Learning With Minimum Description Lengthcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 3; 2026: 2.

11 papers.

  1. 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
  2. 2024Simplicity bias and optimization threshold in two-layer ReLU networkscited by 11
  3. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  4. 2024Provable Tempered Overfitting of Minimal Nets and Typical Netscited by 5
  5. 2025Beyond Benign Overfitting in Nadaraya-Watson Interpolatorscited by 2
  6. 2025Quantifying Overfitting along the Regularization Path for Two-Part-Code MDL in Supervised Classificationcited by 2
  7. 2025Generalization Below the Edge of Stability: The Role of Data Geometrycited by 2
  8. 2024A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Featurescited by 1
  9. 2026On damage of interpolation to adversarial robustness in regressioncited by 1
  10. 2024The Effects of Multi-Task Learning on ReLU Neural Network Functionscited by 0
  11. 2026Harmful Overfitting in Sobolev Spacescited by 0