mlsanityData to 14 Oct 2026

Directional Convergence, Benign Overfitting of Gradient Descent in leaky ReLU two-layer Neural Networks

arXiv 2505.16204 ↗First posted 22 May 2025cs.LG math.ST stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we provide sufficient conditions of benign overfitting of fixed width leaky ReLU two-layer neural network classifiers trained on mixture data via gradient descent. Our results are derived by establishing directional convergence of the network parameters and classification error bound of the convergent direction. Our classification error bound also lead to the discovery of a newly identified phase transition. Previously, directional convergence in (leaky) ReLU neural networks was established only for gradient flow. Due to the lack of directional convergence, previous results on benign overfitting were limited to those trained on nearly orthogonal data. All of our results hold on mixture data, which is a broader data setting than the nearly orthogonal data setting in prior work. We demonstrate our findings by showing that benign overfitting occurs with high probability in a much wider range of scenarios than previously known. Our results also allow us to characterize cases when benign overfitting provably fails even if directional convergence occurs. Our work thus provides a more complete picture of benign overfitting in leaky ReLU two-layer neural networks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 37 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

31 of 37 references.

References · 31

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  6. 2019Benign Overfitting in Linear Regressioncited by 549
  7. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  8. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  10. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  11. 2020Directional convergence and alignment in deep learningcited by 178
  12. 2020Benign overfitting in ridge regressioncited by 153
19 more
  1. 2019Harmless interpolation of noisy data in regressioncited by 121
  2. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  3. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  4. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  5. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  6. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  7. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 49
  8. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  9. 2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
  10. 2020On the proliferation of support vectors in high dimensionscited by 32
  11. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  12. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  13. 2020Binary Classification of Gaussian Mixtures: Abundance of Support Vectors, Benign Overfitting and Regularizationcited by 27
  14. 2021Towards Understanding Learning in Neural Networks with Linear Teacherscited by 19
  15. 2023Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Datacited by 18
  16. 2025Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networkscited by 9
  17. 2024Benign overfitting in leaky ReLU networks with moderate input dimensioncited by 6
  18. 2025Universality of Benign Overfitting in Binary Linear Classificationcited by 3
  19. 2025The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networkscited by 1

Built on by

Not cited yet.