# A Robust Adaptive Stochastic Gradient Method for Deep Learning

arXiv: https://arxiv.org/abs/1703.00788 · HTML: https://mlsanity.com/paper/1703.00788/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2017.

## Built on by (most cited first, 5 of 5)

- [AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization](https://mlsanity.com/paper/2201.06714/) (2022)
- [AWEU-Net: An Attention-Aware Weight Excitation U-Net for Lung Nodule Segmentation](https://mlsanity.com/paper/2110.05144/) (2021)
- [Impact of Learning Rate on Noise Resistant Property of Deep Learning Models](https://mlsanity.com/paper/2205.07856/) (2022)
- [On Unsupervised Image-to-image translation and GAN stability](https://mlsanity.com/paper/2403.09646/) (2023)
- [Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimization](https://mlsanity.com/paper/2411.15795/) (2024)

## Abstract (arXiv)

Stochastic gradient algorithms are the main focus of large-scale optimization problems and led to important successes in the recent advancement of the deep learning algorithms. The convergence of SGD depends on the careful choice of learning rate and the amount of the noise in stochastic estimates of the gradients. In this paper, we propose an adaptive learning rate algorithm, which utilizes stochastic curvature information of the loss function for automatically tuning the learning rates. The information about the element-wise curvature of the loss function is estimated from the local statistics of the stochastic first order gradients. We further propose a new variance reduction technique to speed up the convergence. In our experiments with deep neural networks, we obtained better performance compared to the popular stochastic gradient algorithms.