Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima
Not on the map. Why?
Abstract (arXiv)
An important quantity in the theory of gradient descent (GD) is the sharpness, defined as the largest eigenvalue of the objective Hessian. Classical analyses typically require the step size to be uniformly smaller than twice the reciprocal of the sharpness, but this condition is frequently violated in the training of deep neural networks. Recent work bridges this gap in the setting of overparametrised least-squares with a single scalar output, providing a normal form for large-step GD in a neighbourhood of an isolated flat minimum and establishing three corresponding convergence results. In this paper, we extend this theory in two directions: (1) to overparametrised least-squares with vector-valued outputs (including regression with arbitrarily many observations), and (2) to a neighbourhood of a manifold of flat minima (which we show is essential for applications such as matrix factorisation). We generalise both the normal form and all three convergence theorems of citemacdonaldeos to this broader setting, overcoming several technical challenges. We further show that our framework applies to deep matrix factorisation under mild assumptions, yielding several new structural results. In particular, we prove that the set of flat minima forms a fibre bundle over a product of spheres, and that the sharpness is Morse-Bott along this manifold.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 39 of its 62 references that are in the map. First posted on arXiv 9 Jul 2026.
Builds on
39 of 62 references.
References · 39
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
- 2018On Lazy Training in Differentiable Programmingcited by 675
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
- 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
- 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
- 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
- 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
27 more
- 2022Understanding the unstable convergence of gradient descentcited by 86
- 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
- 2024Understanding Optimization in Deep Learning with Central Flowscited by 64
- 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
- 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
- 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 50
- 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
- 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
- 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
- 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
- 2022Beyond the Edge of Stability via Two-step Gradient Updatescited by 38
- 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
- 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 35
- 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 25
- 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 22
- 2023Provably Faster Gradient Descent via Long Stepscited by 20
- 2023Acceleration by Stepsize Hedging II: Silver Stepsize Schedule for Smooth Convex Optimizationcited by 20
- 2020Flatness is a False Friendcited by 19
- 2023Acceleration by Stepsize Hedging I: Multi-Step Descent and the Silver Stepsize Schedulecited by 19
- 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
- 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
- 2024Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Stepscited by 11
- 2023Accelerated Gradient Descent via Long Stepscited by 9
- 2025Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regressioncited by 9
- 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
- 2025A Minimalist Example of Edge-of-Stability and Progressive Sharpeningcited by 4
- 2025Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squarescited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.