Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent
Not on the map. Why?
Abstract (arXiv)
This paper proposes a novel approach to adaptive step sizes in stochastic gradient descent (SGD) by utilizing quantities that we have identified as numerically traceable -- the Lipschitz constant for gradients and a concept of the local variance in search directions. Our findings yield a nearly hyperparameter-free algorithm for stochastic optimization, which has provable convergence properties and exhibits truly problem adaptive behavior on classical image classification tasks. Our framework is set in a general Hilbert space and thus enables the potential inclusion of a preconditioner through the choice of the inner product.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 17 of its 36 references that are in the map. First posted on arXiv 28 Nov 2023.
Builds on
17 of 36 references.
References · 17
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
- 2016An overview of gradient descent optimization algorithmscited by 1,033
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
- 2014New insights and perspectives on the natural gradient methodcited by 443
- 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
- 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
- 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
- 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
- 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
- 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 119
5 more
- 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 95
- 2019DeepOBS: A Deep Learning Optimizer Benchmark Suitecited by 37
- 2023Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reductioncited by 23
- 2019A Fully Stochastic Second-Order Trust Region Methodcited by 8
- 2023Frobenius-Type Norms and Inner Products of Matrices and Linear Maps with Applications to Neural Network Trainingcited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.