mlsanityData to 7 Oct 2026

Benign Overfitting in Two-layer Convolutional Neural Networks

arXiv 2202.06526 ↗First posted 14 Feb 2022cs.LG math.OC stat.MLcited by 90

Not on the map. Why?

Abstract (arXiv)

Modern neural networks often have great expressive power and can be trained to overfit the training data, while still achieving a good test performance. This phenomenon is referred to as "benign overfitting". Recently, there emerges a line of works studying "benign overfitting" from the theoretical perspective. However, they are limited to linear models or kernel/random feature models, and there is still a lack of theoretical understanding about when and how benign overfitting occurs in neural networks. In this paper, we study the benign overfitting phenomenon in training a two-layer convolutional neural network (CNN). We show that when the signal-to-noise ratio satisfies a certain condition, a two-layer CNN trained by gradient descent can achieve arbitrarily small training and test loss. On the other hand, when this condition does not hold, overfitting becomes harmful and the obtained CNN can only achieve a constant level test loss. These together demonstrate a sharp phase transition between benign overfitting and harmful overfitting, driven by the signal-to-noise ratio. To the best of our knowledge, this is the first work that precisely characterizes the conditions under which benign overfitting can occur in training convolutional neural networks.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Builds on 56 of its 71 references that are in the map. First posted on arXiv 14 Feb 2022.

Builds on

56 of 71 references.

References · 56

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  8. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  9. 2013Network In Networkcited by 600
  10. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  11. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  12. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
38 more
  1. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  2. 2019Benign Overfitting in Linear Regressioncited by 544
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  4. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  5. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  6. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  7. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  8. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  9. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  10. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  11. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  12. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  13. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  14. 2019Two models of double descent for weak featurescited by 194
  15. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  16. 2018To understand deep learning we need to understand kernel learningcited by 186
  17. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  18. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
  19. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  20. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  21. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  22. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  23. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  24. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  25. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  26. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  27. 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
  28. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  29. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  30. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  31. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  32. 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 69
  33. 2017Learning ReLUs via Gradient Descentcited by 54
  34. 2018Stability and Convergence Trade-off of Iterative Optimization Algorithmscited by 53
  35. 2018Learning One-hidden-layer ReLU Networks via Gradient Descentcited by 50
  36. 2020Algorithms and SQ Lower Bounds for PAC Learning One-Hidden-Layer ReLU Networkscited by 49
  37. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  38. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 48

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 28; 2024: 21; 2025: 19; 2026: 10.

90 papers.

Sort:

By area:

  1. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  2. 2025Deep Learning is Not So Mysterious or Differentcited by 55
  3. 2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
  4. 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
  5. 2023On the Role of Attention in Prompt-tuningcited by 49
  6. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  7. 2023Robust Learning with Progressive Data Expansion Against Spurious Correlationcited by 39
  8. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  9. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  10. 2023Understanding and Improving Feature Learning for Out-of-Distribution Generalizationcited by 35
  11. 2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
  12. 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
38 more
  1. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  2. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  3. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 27
  4. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
  5. 2023Generalization and Stability of Interpolating Neural Networks with Minimal Widthcited by 21
  6. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Methodcited by 21
  7. 2023The Benefits of Mixup for Feature Learningcited by 19
  8. 2023Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Datacited by 18
  9. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  10. 2023Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Modelscited by 14
  11. 2024Memorization in deep learning: A surveycited by 14
  12. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  13. 2024GenMix: Effective Data Augmentation with Generative Diffusion Model Image Editingcited by 14
  14. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  15. 2023Quantifying the Optimization and Generalization Advantages of Graph Neural Networks Over Multilayer Perceptronscited by 12
  16. 2022Understanding Benign Overfitting in Gradient-Based Meta Learningcited by 11
  17. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  18. 2024Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learningcited by 11
  19. 2023Benign Overfitting in Deep Neural Networks under Lazy Trainingcited by 10
  20. 2023Optimal Rate of Kernel Regression in Large Dimensionscited by 10
  21. 2023Benign Overfitting in Two-Layer ReLU Convolutional Neural Networks for XOR Datacited by 10
  22. 2022Provable Generalization of Overparameterized Meta-learning Trained with SGDcited by 9
  23. 2022Learning Ability of Interpolating Deep Convolutional Neural Networkscited by 9
  24. 2024NLPrompt: Noise-Label Prompt Learning for Vision-Language Modelscited by 9
  25. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
  26. 2023The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networkscited by 8
  27. 2022A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictorscited by 7
  28. 2023Per-Example Gradient Regularization Improves Learning Signals from Noisy Datacited by 7
  29. 2023Understanding the Initial Condensation of Convolutional Neural Networkscited by 7
  30. 2023Training shallow ReLU networks on noisy data using hinge loss: when do we overfit and is it benign?cited by 7
  31. 2024On the Benefits of Over-parameterization for Out-of-Distribution Generalizationcited by 7
  32. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  33. 2024DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticitycited by 7
  34. 2022Max-Margin Works while Large Margin Fails: Generalization without Uniform Convergencecited by 6
  35. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 6
  36. 2022Benign Overfitting in Classification: Provably Counter Label Noise with Larger Modelscited by 5
  37. 2024Benign overfitting in leaky ReLU networks with moderate input dimensioncited by 5
  38. 2024Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samplescited by 5

50 most cited of 90 All 90 (JSON)