mlsanityData to 14 Oct 2026

Benign overfitting in leaky ReLU networks with moderate input dimension

arXiv 2403.06903 ↗First posted 11 Mar 2024cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

The problem of benign overfitting asks whether it is possible for a model to perfectly fit noisy training data and still generalize well. We study benign overfitting in two-layer leaky ReLU networks trained with the hinge loss on a binary classification task. We consider input data that can be decomposed into the sum of a common signal and a random noise component, that lie on subspaces orthogonal to one another. We characterize conditions on the signal to noise ratio (SNR) of the model parameters giving rise to benign versus non-benign (or harmful) overfitting: in particular, if the SNR is high then benign overfitting occurs, conversely if the SNR is low then harmful overfitting occurs. We attribute both benign and non-benign overfitting to an approximate margin maximization property and show that leaky ReLU networks trained on hinge loss with gradient descent (GD) satisfy this property. In contrast to prior work we do not require the training data to be nearly orthogonal. Notably, for input dimension d and training sample size n, while results in prior work require d = Omega(n2 log n), here we require only d = Omegaleft(nright).

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

31 of 39 references.

References · 31

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2019Benign Overfitting in Linear Regressioncited by 549
  4. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  6. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  7. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  8. 2018To understand deep learning we need to understand kernel learningcited by 187
  9. 2020Directional convergence and alignment in deep learningcited by 178
  10. 2019Harmless interpolation of noisy data in regressioncited by 121
  11. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  12. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 98
19 more
  1. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  2. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  3. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  4. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  5. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  6. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  7. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  8. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 49
  9. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 42
  10. 2022The Implicit Bias of Benign Overfittingcited by 38
  11. 2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
  12. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 33
  13. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  14. 2019The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regimecited by 28
  15. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 28
  16. 2021Foolish Crowds Support Benign Overfittingcited by 24
  17. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  18. 2023Training shallow ReLU networks on noisy data using hinge loss: when do we overfit and is it benign?cited by 7
  19. 2023Universality of max-margin classifierscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 3.

6 papers.

  1. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  2. 2024Provable Tempered Overfitting of Minimal Nets and Typical Netscited by 5
  3. 2026Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learningcited by 3
  4. 2026Harmful Overfitting in Sobolev Spacescited by 1
  5. 2025Directional Convergence, Benign Overfitting of Gradient Descent in leaky ReLU two-layer Neural Networkscited by 0
  6. 2026Double descent for least-squares interpolation on contaminated data: A simulation studycited by 0