Learning-Rate-Free Learning: Dissecting D-Adaptation and Probabilistic Line Search
Not on the map. Why?
Abstract (arXiv)
This paper explores two recent methods for learning rate optimisation in stochastic gradient descent: D-Adaptation (arXiv:2301.07733) and probabilistic line search (arXiv:1502.02846). These approaches aim to alleviate the burden of selecting an initial learning rate by incorporating distance metrics and Gaussian process posterior estimates, respectively. In this report, I provide an intuitive overview of both methods, discuss their shared design goals, and devise scope for merging the two algorithms.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 5 of its 13 references that are in the map. First posted on arXiv 6 Aug 2023.
Builds on
5 of 13 references.
References · 5
- 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
- 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
- 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
- 2020Recent Theoretical Advances in Non-Convex Optimizationcited by 30
- 2015Probabilistic Line Searches for Stochastic Optimizationcited by 29
Built on by
Not cited yet.