mlsanityData to 7 Oct 2026

Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization

arXiv 2406.08654 ↗First posted 12 Jun 2024stat.ML cs.LG math.OCcited by 17

Not on the map. Why?

Abstract (arXiv)

The typical training of neural networks using large stepsize gradient descent (GD) under the logistic loss often involves two distinct phases, where the empirical risk oscillates in the first phase but decreases monotonically in the second phase. We investigate this phenomenon in two-layer networks that satisfy a near-homogeneity condition. We show that the second phase begins once the empirical risk falls below a certain threshold, dependent on the stepsize. Additionally, we show that the normalized margin grows nearly monotonically in the second phase, demonstrating an implicit bias of GD in training non-homogeneous predictors. If the dataset is linearly separable and the derivative of the activation function is bounded away from zero, we show that the average empirical risk decreases, implying that the first phase must stop in finite steps. Finally, we demonstrate that by choosing a suitably large stepsize, GD that undergoes this phase transition is more efficient than GD that monotonically decreases the risk. Our analysis applies to networks of any width, beyond the well-known neural tangent kernel and mean-field regimes.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 41 of its 47 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

41 of 47 references.

References · 41

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  7. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  8. 2017Implicit Regularization in Matrix Factorizationcited by 318
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  10. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  11. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  12. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
29 more
  1. 2020Directional convergence and alignment in deep learningcited by 172
  2. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  3. 2020Implicit Gradient Regularizationcited by 145
  4. 2018Convergence of Gradient Descent on Separable Datacited by 108
  5. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  6. 2022Understanding the unstable convergence of gradient descentcited by 84
  7. 2018Risk and parameter convergence of logistic regressioncited by 81
  8. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  9. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  10. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 63
  11. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  12. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  13. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  14. 2022Learning threshold neurons via the "edge of stability"cited by 50
  15. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  16. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 42
  17. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  18. 2022Beyond the Edge of Stability via Two-step Gradient Updatescited by 38
  19. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 34
  20. 2021Fast Margin Maximization via Dual Accelerationcited by 33
  21. 2020Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Functioncited by 26
  22. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
  23. 2022Uniform-in-time propagation of chaos for mean field Langevin dynamicscited by 23
  24. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 23
  25. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 20
  26. 2023Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reductioncited by 19
  27. 2021Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noisecited by 17
  28. 2021When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?cited by 9
  29. 2024Understanding the Generalization Benefits of Late Learning Rate Decaycited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 6.

17 papers.

Sort:

By area:

  1. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 13
  2. 2025Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regressioncited by 7
  3. 2025Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizescited by 5
  4. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 5
  5. 2025Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squarescited by 3
  6. 2025Statistical Guarantees for High-Dimensional Stochastic Gradient Descentcited by 2
  7. 2025From Information to Generative Exponent: Learning Rate Induces Phase Transitions in SGDcited by 2
  8. 2026SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMscited by 2
  9. 2025Any-stepsize Gradient Descent for Separable Data under Fenchel-Young Lossescited by 1
  10. 2025The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networkscited by 1
  11. 2025Constant Stepsize Local GD for Logistic Regression: Acceleration by Instabilitycited by 1
  12. 2026Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stabilitycited by 1
5 more
  1. 2026Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minimacited by 1
  2. 2025Embedding principle of homogeneous neural network for classification problemcited by 0
  3. 2026Tight Bounds for Logistic Regression with Large Stepsize Gradient Descent in Low Dimensioncited by 0
  4. 2026Centre manifold theorem for maps along manifolds of fixed pointscited by 0
  5. 2026The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flowcited by 0