mlsanityData to 14 Oct 2026

From Tempered to Benign Overfitting in ReLU Neural Networks

arXiv 2305.15141 ↗First posted 24 May 2023cs.LG cs.NE stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

Overparameterized neural networks (NNs) are observed to generalize well even when trained to perfectly fit noisy data. This phenomenon motivated a large body of work on "benign overfitting", where interpolating predictors achieve near-optimal performance. Recently, it was conjectured and empirically observed that the behavior of NNs is often better described as "tempered overfitting", where the performance is non-optimal yet also non-trivial, and degrades as a function of the noise level. However, a theoretical justification of this claim for non-linear NNs has been lacking so far. In this work, we provide several results that aim at bridging these complementing views. We study a simple classification setting with 2-layer ReLU NNs, and prove that under various assumptions, the type of overfitting transitions from tempered in the extreme case of one-dimensional data, to benign in high dimensions. Thus, we show that the input dimension has a crucial role on the type of overfitting in this setting, which we also validate empirically for intermediate dimensions. Overall, our results shed light on the intricate connections between the dimension, sample size, architecture and training algorithm on the one hand, and the type of resulting overfitting on the other hand.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 44 of its 59 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

44 of 59 references.

References · 44

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  4. 2019Benign Overfitting in Linear Regressioncited by 549
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  7. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  8. 2021Deep learning: a statistical viewpointcited by 240
  9. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 224
  10. 2019Two models of double descent for weak featurescited by 195
  11. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  12. 2018To understand deep learning we need to understand kernel learningcited by 187
32 more
  1. 2020Directional convergence and alignment in deep learningcited by 178
  2. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
  3. 2019How do infinite width bounded norm networks look in function space?cited by 106
  4. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  5. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  6. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  7. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  8. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  9. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  10. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  11. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 49
  12. 2018Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenoncited by 44
  13. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  14. 2020Distributional Generalization: A New Kind of Generalizationcited by 42
  15. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 42
  16. 2022The Implicit Bias of Benign Overfittingcited by 38
  17. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  18. 2020Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic Viewcited by 32
  19. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  20. 2020Failures of model-dependent generalization bounds for least-norm interpolationcited by 26
  21. 2021Interpolating Classifiers Make Few Mistakescited by 22
  22. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  23. 2023Penalising the biases in norm regularisation enforces sparsitycited by 19
  24. 2021Optimistic Rates: A Unifying Theory for Interpolation Learning and Regularization in Linear Regressioncited by 17
  25. 2021Harmless interpolation in regression and classification with structured featurescited by 14
  26. 2021Understanding Square Loss in Training Overparametrized Neural Network Classifierscited by 14
  27. 2021Exact Gap between Generalization Error and Uniform Convergence in Random Feature Modelscited by 13
  28. 2022On the Inconsistency of Kernel Ridgeless Regression in Fixed Dimensionscited by 13
  29. 2021Uniform Convergence, Adversarial Spheres and a Simple Remedycited by 8
  30. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
  31. 2022Max-Margin Works while Large Margin Fails: Generalization without Uniform Convergencecited by 6
  32. 2023Interpolation Learning With Minimum Description Lengthcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 4; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 33
  2. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 14
  3. 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
  4. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  5. 2023The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Modelscited by 7
  6. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  7. 2024Benign overfitting in leaky ReLU networks with moderate input dimensioncited by 6
  8. 2024Provable Tempered Overfitting of Minimal Nets and Typical Netscited by 5
  9. 2024Neural Networks Generalize on Low Complexity Datacited by 3
  10. 2025Scanning Trojaned Models Using Out-of-Distribution Samplescited by 3
  11. 2025Generalization Below the Edge of Stability: The Role of Data Geometrycited by 3
  12. 2025Beyond Benign Overfitting in Nadaraya-Watson Interpolatorscited by 2
4 more
  1. 2025Quantifying Overfitting along the Regularization Path for Two-Part-Code MDL in Supervised Classificationcited by 2
  2. 2024A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Featurescited by 1
  3. 2026Harmful Overfitting in Sobolev Spacescited by 1
  4. 2026Neuron Block Dynamics for XOR Classification with Zero-Margincited by 0