mlsanityData to 7 Oct 2026

Benign Overfitting for Two-layer ReLU Convolutional Neural Networks

arXiv 2303.04145 ↗First posted 7 Mar 2023cs.LG math.OC stat.MLcited by 34

Not on the map. Why?

Abstract (arXiv)

Modern deep learning models with great expressive power can be trained to overfit the training data but still generalize well. This phenomenon is referred to as textitbenign overfitting. Recently, a few studies have attempted to theoretically understand benign overfitting in neural networks. However, these works are either limited to neural networks with smooth activation functions or to the neural tangent kernel regime. How and when benign overfitting can occur in ReLU neural networks remains an open problem. In this work, we seek to answer this question by establishing algorithm-dependent risk bounds for learning two-layer ReLU convolutional neural networks with label-flipping noise. We show that, under mild conditions, the neural network trained by gradient descent can achieve near-zero training loss and Bayes optimal test risk. Our result also reveals a sharp transition between benign and harmful overfitting under different conditions on data distribution in terms of test risk. Experiments on synthetic data back up our theory.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 34 of its 44 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

34 of 44 references.

References · 34

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  5. 2013Network In Networkcited by 600
  6. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  7. 2019Benign Overfitting in Linear Regressioncited by 544
  8. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  9. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  10. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  11. 2019Two models of double descent for weak featurescited by 194
  12. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
22 more
  1. 2018To understand deep learning we need to understand kernel learningcited by 186
  2. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  3. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
  4. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  5. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  6. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  7. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  8. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  9. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  10. 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
  11. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  12. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  13. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  14. 2018Stability and Convergence Trade-off of Iterative Optimization Algorithmscited by 53
  15. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  16. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 48
  17. 2022The Implicit Bias of Benign Overfittingcited by 38
  18. 2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 29
  19. 2021Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descentcited by 21
  20. 2022Data Augmentation as Feature Manipulationcited by 18
  21. 2021Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noisecited by 17
  22. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 9; 2025: 14; 2026: 8.

34 papers.

Sort:

By area:

  1. 2025Deep Learning is Not So Mysterious or Differentcited by 55
  2. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  3. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  4. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Methodcited by 21
  5. 2023Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Datacited by 18
  6. 2024GenMix: Effective Data Augmentation with Generative Diffusion Model Image Editingcited by 14
  7. 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
  8. 2024On the Comparison between Multi-modal and Single-modal Contrastive Learningcited by 10
  9. 2023Per-Example Gradient Regularization Improves Learning Signals from Noisy Datacited by 7
  10. 2024Benign overfitting in leaky ReLU networks with moderate input dimensioncited by 5
  11. 2024Provable Tempered Overfitting of Minimal Nets and Typical Netscited by 5
  12. 2025On the Role of Label Noise in the Feature Learning Processcited by 4
22 more
  1. 2025How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?cited by 4
  2. 2024Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalizationcited by 3
  3. 2025Scanning Trojaned Models Using Out-of-Distribution Samplescited by 3
  4. 2023Membership Privacy Risks of Sharpness Aware Minimizationcited by 2
  5. 2025Rethinking Benign Overfitting in Two-Layer Neural Networkscited by 2
  6. 2025Understanding Generalization in Transformers: Error Bounds and Training Dynamics Under Benign and Harmful Overfittingcited by 2
  7. 2025Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Modelscited by 2
  8. 2025Understanding Private Learning From Feature Perspectivecited by 2
  9. 2025Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Datacited by 1
  10. 2026Provable Effects of Data Replay in Continual Learning: A Feature Learning Perspectivecited by 1
  11. 2024Bayesian Inference for Consistent Predictions in Overparameterized Nonlinear Regressioncited by 0
  12. 2025Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networkscited by 0
  13. 2025Directional Convergence, Benign Overfitting of Gradient Descent in leaky ReLU two-layer Neural Networkscited by 0
  14. 2025FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Promptscited by 0
  15. 2025DFedReweighting: A Unified Framework for Objective-Oriented Reweighting in Decentralized Federated Learningcited by 0
  16. 2026Deep Exploration of Epoch-wise Double Descent in Noisy Data: Signal Separation, Large Activation, and Benign Overfittingcited by 0
  17. 2026Neuron Block Dynamics for XOR Classification with Zero-Margincited by 0
  18. 2026Harmful Overfitting in Sobolev Spacescited by 0
  19. 2026Why Some Models Resist Unlearning: A Linear Stability Perspectivecited by 0
  20. 2026Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Datacited by 0
  21. 2026When Does ℓ2-Boosting Overfit Benignly? High-Dimensional Risk Asymptotics and the ℓ1 Implicit Biascited by 0
  22. 2026Signal-Noise Factorization Isolates Nuisance Variation into Removable Subspacescited by 0