Data to 14 Oct 2026

Variants of RMSProp and Adagrad with Logarithmic Regret Bounds

arXiv 1706.05507 ↗First posted 17 Jun 2017cs.LG cs.AI cs.CVcited by 50

Not on the map. Why?

Abstract (arXiv)

Adaptive gradient methods have become recently very popular, in particular as they have been shown to be useful in the training of deep neural networks. In this paper we have analyzed RMSProp, originally proposed for the training of deep neural networks, in the context of online convex optimization and show √T-type regret bounds. Moreover, we propose two variants SC-Adagrad and SC-RMSProp for which we show logarithmic regret bounds for strongly convex functions. Finally, we demonstrate in the experiments that these new variants outperform other adaptive gradient techniques or stochastic gradient descent in the optimization of strongly convex functions as well as in training of deep neural networks.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 6; 2023: 10; 2024: 9; 2025: 4; 2026: 6.

50 papers.

Sort:

By area:

2021SUPER-ADAM: Faster and Universal Framework of Adaptive Gradientscited by 40
2022Private Adaptive Optimization with Side Informationcited by 38
2021Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Accelerationcited by 25
2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
2023Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reductioncited by 23
2024AdaGrad under Anisotropic Smoothnesscited by 22
2022Nest Your Adaptive Algorithm for Parameter-Agnostic Nonconvex Minimax Optimizationcited by 16
2024An Improved Empirical Fisher Approximation for Natural Gradient Descentcited by 15
2023Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Casecited by 14
2023AnalogNAS: A Neural Network Design Framework for Accurate Inference with Analog In-Memory Computingcited by 10
2021Adam revisited: a weighted past gradients perspectivecited by 9
2021Efficient Semi-Implicit Variational Inferencecited by 9

38 more

2021On the One-sided Convergence of Adam-type Algorithms in Non-convex Non-concave Min-max Optimizationcited by 9
2021The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methodscited by 7
2021KOALA: A Kalman Optimization Algorithm with Loss Adaptivitycited by 6
2023Boosting Diffusion Models with an Adaptive Momentum Samplercited by 6
2023Adaptive, Doubly Optimal No-Regret Learning in Strongly Monotone and Exp-Concave Games with Gradient Feedbackcited by 6
2024Universal Online Convex Optimization with 1 Projection per Roundcited by 6
2021Universal Online Convex Optimization Meets Second-order Boundscited by 4
2021AdaLoss: A computationally-efficient and provably convergent adaptive gradient methodcited by 4
2022Dynamic Regret of Adaptive Gradient Methods for Strongly Convex Problemscited by 4
2023Skeleton-based Human Action Recognition via Convolutional Neural Networks (CNN)cited by 4
2024DCNet: A Data-Driven Framework for DVL Calibrationcited by 4
2021Multi-Party Dual Learningcited by 3
2022A Stochastic Bundle Method for Interpolating Networkscited by 3
2023Bridging Classical and Quantum Machine Learning: Knowledge Transfer From Classical to Quantum Neural Networks Using Knowledge Distillationcited by 3
2023FedDA: Faster Framework of Local Adaptive Gradient Methods via Restarted Dual Averagingcited by 2
2024Probabilistic Neural Networks (PNNs) for Modeling Aleatoric Uncertainty in Scientific Machine Learningcited by 2
2024Towards Simple and Provable Parameter-Free Adaptive Gradient Methodscited by 2
2025Preconditioned Inexact Stochastic ADMM for Deep Modelcited by 2
2026A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad and Muoncited by 2
2021Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learningcited by 1
2025A Diagnostic Evaluation of Neural Networks Trained with the Error Diffusion Learning Algorithmcited by 1
2026The Power of Decaying Steps: Enhancing Attack Stability and Transferability for Sign-based Optimizerscited by 1
2026Adaptivity via a Parallel Architecture for Stochastic Gradient Methodscited by 1
2021Training Deep Neural Networks via Branch-and-Boundcited by 0
2021Meta-Regularization: An Approach to Adaptive Choice of the Learning Rate in Gradient Descentcited by 0
2021FastAdaBelief: Improving Convergence Rate for Belief-based Adaptive Optimizers by Exploiting Strong Convexitycited by 0
2021Mixing between the Cross Entropy and the Expectation Loss Termscited by 0
2022A Learning and Control Perspective for Microfinancecited by 0
2022Comparision Of Adversarial And Non-Adversarial LSTM Music Generative Modelscited by 0
2023Estimation of Sea State Parameters from Ship Motion Responses Using Attention-based Neural Networkscited by 0
2023Parameter-free version of Adaptive Gradient Methods for Strongly-Convex Functionscited by 0
2024A Selective Review on Statistical Methods for Massive Data Computation: Distributed Computing, Subsampling, and Minibatch Techniquescited by 0
2024An Automated SQL Query Grading System Using An Attention-Based Convolutional Neural Networkcited by 0
2025Adaptive Moment Estimation Optimization Algorithm Using Projection Gradient for Deep Learningcited by 0
2025Optimizing the Adversarial Perturbation with a Momentum-based Adaptive Matrixcited by 0
2026Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorizationcited by 0
2026Convergence rates for the RMSprop optimizer with full control of the hyperparameterscited by 0
2026Revisiting AdaGrad in Stochastic Convex Optimization: Last Iterates, High Probability, and Lower Boundscited by 0