mlsanityData to 7 Oct 2026

Benign Overfitting in Classification: Provably Counter Label Noise with Larger Models

arXiv 2206.00501 ↗First posted 1 Jun 2022cs.LG cs.AI stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Studies on benign overfitting provide insights for the success of overparameterized deep learning models. In this work, we examine whether overfitting is truly benign in real-world classification tasks. We start with the observation that a ResNet model overfits benignly on Cifar10 but not benignly on ImageNet. To understand why benign overfitting fails in the ImageNet experiment, we theoretically analyze benign overfitting under a more restrictive setup where the number of parameters is not significantly larger than the number of data points. Under this mild overparameterization setup, our analysis identifies a phase change: unlike in the previous heavy overparameterization settings, benign overfitting can now fail in the presence of label noise. Our analysis explains our empirical observations, and is validated by a set of control experiments with ResNets. Our work highlights the importance of understanding implicit bias in underfitting regimes as a future direction.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 37 of its 57 references that are in the map. First posted on arXiv 1 Jun 2022.

Builds on

37 of 57 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  8. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  9. 2019Benign Overfitting in Linear Regressioncited by 544
  10. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  11. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  12. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
25 more
  1. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  2. 2021Deep learning: a statistical viewpointcited by 239
  3. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  4. 2018To understand deep learning we need to understand kernel learningcited by 186
  5. 2020Benign overfitting in ridge regressioncited by 152
  6. 2020Self-Adaptive Training: beyond Empirical Risk Minimizationcited by 126
  7. 2019Harmless interpolation of noisy data in regressioncited by 121
  8. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  9. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 117
  10. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  11. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  12. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  13. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  14. 2018A modern maximum-likelihood theory for high-dimensional logistic regressioncited by 79
  15. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  16. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  17. 2018Label Refinery: Improving ImageNet Classification through Label Progressioncited by 65
  18. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 63
  19. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  20. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  21. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 41
  22. 2020Early Stopping in Deep Networks: Double Descent and How to Eliminate itcited by 40
  23. 2021SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochscited by 22
  24. 2021When and how epochwise double descent happenscited by 20
  25. 2019High Dimensional Classification via Regularized and Unregularized Empirical Risk Minimization: Precise Error and Optimal Losscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 1; 2026: 1.

5 papers.

  1. 2023Understanding and Mitigating the Label Noise in Pre-training on Downstream Taskscited by 27
  2. 2024Impact of Noisy Supervision in Foundation Model Learningcited by 8
  3. 2025Shifting the Paradigm: A Diffeomorphism Between Time Series Data Manifolds for Achieving Shift-Invariancy in Deep Learningcited by 5
  4. 2024Coordinated Sparse Recovery of Label Noisecited by 0
  5. 2026Generalization Bounds of Stochastic Gradient Descent in Homogeneous Neural Networkscited by 0