Rethinking Benign Overfitting in Two-Layer Neural Networks
Not on the map. Why?
Abstract (arXiv)
Recent theoretical studies (Kou et al., 2023; Cao et al., 2022) have revealed a sharp phase transition from benign to harmful overfitting when the noise-to-feature ratio exceeds a threshold-a situation common in long-tailed data distributions where atypical data is prevalent. However, harmful overfitting rarely happens in overparameterized neural networks. Further experimental results suggested that memorization is necessary for achieving near-optimal generalization error in long-tailed data distributions (Feldman & Zhang, 2020). We argue that this discrepancy between theoretical predictions and empirical observations arises because previous feature-noise data models overlook the heterogeneous nature of noise across different data classes. In this paper, we refine the feature-noise data model by incorporating class-dependent heterogeneous noise and re-examine the overfitting phenomenon in neural networks. Through a comprehensive analysis of the training dynamics, we establish test loss bounds for the refined model. Our findings reveal that neural networks can leverage "data noise" to learn implicit features that improve the classification accuracy for long-tailed data. Our analysis also provides a training-free metric for evaluating data influence on test performance. Experimental validation on both synthetic and real-world datasets supports our theoretical results.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 26 references that are in the map. First posted on arXiv 17 Feb 2025.
Builds on
22 of 26 references.
References · 22
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
- 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
- 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
- 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
- 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
- 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 104
- 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
10 more
- 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
- 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
- 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
- 2022Towards understanding how momentum improves generalization in deep learningcited by 40
- 2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
- 2023Memorization Through the Lens of Curvature of Loss Function Around Samplescited by 27
- 2023The Benefits of Mixup for Feature Learningcited by 19
- 2024Memorization in Self-Supervised Learning Improves Downstream Generalizationcited by 18
- 2022Memorize to Generalize: on the Necessity of Interpolation in High Dimensional Linear Regressioncited by 14
- 2022Measuring Unintended Memorisation of Unique Private Features in Neural Networkscited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.