mlsanityData to 7 Oct 2026

Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Data

arXiv 2202.05928 ↗First posted 11 Feb 2022cs.LG math.ST stat.MLcited by 74

Not on the map. Why?

Abstract (arXiv)

Benign overfitting, the phenomenon where interpolating models generalize well in the presence of noisy data, was first observed in neural network models trained with gradient descent. To better understand this empirical observation, we consider the generalization error of two-layer neural networks trained to interpolation by gradient descent on the logistic loss following random initialization. We assume the data comes from well-separated class-conditional log-concave distributions and allow for a constant fraction of the training labels to be corrupted by an adversary. We show that in this setting, neural networks exhibit benign overfitting: they can be driven to zero training error, perfectly fitting any noisy training labels, and simultaneously achieve minimax optimal test error. In contrast to previous work on benign overfitting that require linear or kernel-based predictors, our analysis holds in a setting where both the model and learning dynamics are fundamentally nonlinear.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 11 Feb 2022.

Builds on

40 of 56 references.

References · 40

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  8. 2019Benign Overfitting in Linear Regressioncited by 544
  9. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  10. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  11. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  12. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
28 more
  1. 2020Benign overfitting in ridge regressioncited by 152
  2. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  3. 2019Harmless interpolation of noisy data in regressioncited by 121
  4. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  5. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  6. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  7. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  8. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  9. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  10. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  11. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  12. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  13. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  14. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 41
  15. 2021Early-stopped neural networks are consistentcited by 34
  16. 2020On the proliferation of support vectors in high dimensionscited by 32
  17. 2020Binary Classification of Gaussian Mixtures: Abundance of Support Vectors, Benign Overfitting and Regularizationcited by 27
  18. 2018Partial recovery bounds for clustering with the relaxed Kmeanscited by 25
  19. 2021Foolish Crowds Support Benign Overfittingcited by 24
  20. 2021Interpolating Classifiers Make Few Mistakescited by 22
  21. 2021Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descentcited by 21
  22. 2021Is Importance Weighting Incompatible with Interpolating Classifiers?cited by 19
  23. 2020On the robustness of minimum norm interpolators and regularized empirical risk minimizerscited by 18
  24. 2021The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networkscited by 18
  25. 2021Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noisecited by 17
  26. 2019Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networkscited by 16
  27. 2021Tight bounds for minimum l1-norm interpolation of noisy datacited by 16
  28. 2021Minimax Supervised Clustering in the Anisotropic Gaussian Mixture Model: A new take on Robust Interpolationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 19; 2023: 23; 2024: 16; 2025: 13; 2026: 3.

74 papers.

Sort:

By area:

  1. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  2. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  3. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  4. 2024How Does Unlabeled Data Provably Help Out-of-Distribution Detection?cited by 36
  5. 2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
  6. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 33
  7. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
  8. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  9. 2023On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domainscited by 30
  10. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  11. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
  12. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 27
38 more
  1. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  2. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
  3. 2023Generalization and Stability of Interpolating Neural Networks with Minimal Widthcited by 21
  4. 2023The Benefits of Mixup for Feature Learningcited by 19
  5. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  6. 2024Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Contextcited by 14
  7. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 13
  8. 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
  9. 2022Understanding Benign Overfitting in Gradient-Based Meta Learningcited by 11
  10. 2022The Curious Case of Benign Memorizationcited by 11
  11. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  12. 2022Multiple Descent in the Multiple Random Feature Modelcited by 10
  13. 2022Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processescited by 10
  14. 2023Benign Overfitting in Deep Neural Networks under Lazy Trainingcited by 10
  15. 2023Benign Overfitting in Two-Layer ReLU Convolutional Neural Networks for XOR Datacited by 10
  16. 2022Mitigating multiple descents: A model-agnostic framework for risk monotonizationcited by 9
  17. 2025Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictionscited by 9
  18. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
  19. 2022A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictorscited by 7
  20. 2023Per-Example Gradient Regularization Improves Learning Signals from Noisy Datacited by 7
  21. 2023Training shallow ReLU networks on noisy data using hinge loss: when do we overfit and is it benign?cited by 7
  22. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  23. 2022Max-Margin Works while Large Margin Fails: Generalization without Uniform Convergencecited by 6
  24. 2022Benign Overfitting in Classification: Provably Counter Label Noise with Larger Modelscited by 5
  25. 2023On the Clean Generalization and Robust Overfitting in Adversarial Training from Two Theoretical Views: Representation Complexity and Training Dynamicscited by 5
  26. 2024Benign overfitting in leaky ReLU networks with moderate input dimensioncited by 5
  27. 2024Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samplescited by 5
  28. 2024Provable Tempered Overfitting of Minimal Nets and Typical Netscited by 5
  29. 2025Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Modelscited by 5
  30. 2022On how to avoid exacerbating spurious correlations when models are overparameterizedcited by 4
  31. 2022GPT-Neo for commonsense reasoning -- a theoretical and practical lenscited by 4
  32. 2023General Loss Functions Lead to (Approximate) Interpolation in High Dimensionscited by 4
  33. 2023Universality of max-margin classifierscited by 4
  34. 2023Analysis of the expected L2 error of an over-parametrized deep neural network estimate learned by gradient descent without regularizationcited by 4
  35. 2024Provable Privacy Attacks on Trained Shallow Neural Networkscited by 4
  36. 2024Provable Benefit of Cutout and CutMix for Feature Learningcited by 4
  37. 2025On the Role of Label Noise in the Feature Learning Processcited by 4
  38. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4

50 most cited of 74 All 74 (JSON)