A Full Adagrad algorithm with O(Nd) operations
Not on the map. Why?
Abstract (arXiv)
A novel approach is given to overcome the computational challenges of the full-matrix Adaptive Gradient algorithm (Full AdaGrad) in stochastic optimization. By developing a recursive method that estimates the inverse of the square root of the covariance of the gradient, alongside a streaming variant for parameter updates, the study offers efficient and practical algorithms for large-scale applications. This innovative strategy significantly reduces the complexity and resource demands typically associated with full-matrix methods, enabling more effective optimization processes. Moreover, the convergence rates of the proposed estimators and their asymptotic efficiency are given. Their effectiveness is demonstrated through numerical studies.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 9 of its 25 references that are in the map. First posted on arXiv 3 May 2024.
Builds on
9 of 25 references.
References · 9
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
- 2016An overview of gradient descent optimization algorithmscited by 1,021
- 2015Cyclical Learning Rates for Training Neural Networkscited by 627
- 2016Stochastic Optimization for Large-scale Optimal Transportcited by 188
- 2020A Simple Convergence Proof of Adam and Adagradcited by 169
- 2019A Survey of Optimization Methods from a Machine Learning Perspectivecited by 90
- 2020Online Covariance Matrix Estimation in Stochastic Gradient Descentcited by 37
- 2023Online stochastic Newton methods for estimating the geometric median and applicationscited by 2
- 2024Online estimation of the inverse of the Hessian for stochastic optimization with application to universal stochastic Newton algorithmscited by 2
Built on by
Not cited yet.