# Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications

arXiv: https://arxiv.org/abs/2303.01370 · HTML: https://mlsanity.com/paper/2303.01370/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 29 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Builds on (most cited first, 9 of 9)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [SGD: General Analysis and Improved Rates](https://mlsanity.com/paper/1901.09401/) (2019)
- [A Simple Convergence Proof of Adam and Adagrad](https://mlsanity.com/paper/2003.02395/) (2020)
- [Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)](https://mlsanity.com/paper/1306.2119/) (2013)
- [Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression](https://mlsanity.com/paper/1303.6149/) (2013)
- [Asymptotic study of stochastic adaptive algorithm in non-convex landscape](https://mlsanity.com/paper/2012.05640/) (2020)
- [Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Setting](https://mlsanity.com/paper/1504.04407/) (2015)
- [Stochastic Online Optimization using Kalman Recursion](https://mlsanity.com/paper/2002.03636/) (2020)
- [Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Streaming Data](https://mlsanity.com/paper/2109.07117/) (2021)

## Built on by (most cited first, 7 of 7)

- [Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates](https://mlsanity.com/paper/2407.08100/) (2024)
- [Convergence rates for the Adam optimizer](https://mlsanity.com/paper/2407.21078/) (2024)
- [Sharp higher order convergence rates for the Adam optimizer](https://mlsanity.com/paper/2504.19426/) (2025)
- [Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation](https://mlsanity.com/paper/2402.02857/) (2024)
- [Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks](https://mlsanity.com/paper/2607.04233/) (2026)
- [Inversion-Free Natural Gradient Descent on Riemannian Manifolds](https://mlsanity.com/paper/2604.02969/) (2026)
- [Convergence rates for the RMSprop optimizer with full control of the hyperparameters](https://mlsanity.com/paper/2608.30382/) (2026)

## Abstract (arXiv)

In stochastic optimization, a common tool to deal sequentially with large sample is to consider the well-known stochastic gradient algorithm. Nevertheless, since the stepsequence is the same for each direction, this can lead to bad results in practice in case of ill-conditionned problem. To overcome this, adaptive gradient algorithms such that Adagrad or Stochastic Newton algorithms should be prefered. This paper is devoted to the non asymptotic analyis of these adaptive gradient algorithms for strongly convex objective. All the theoretical results will be adapted to linear regression and regularized generalized linear model for both Adagrad and Stochastic Newton algorithms.