# AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio

arXiv: https://arxiv.org/abs/2204.00825 · HTML: https://mlsanity.com/paper/2204.00825/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 30 references that are in the map.
- First posted on arXiv 2 Apr 2022.

## Builds on (most cited first, 9 of 9)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [Matrix Decomposition and Applications](https://mlsanity.com/paper/2201.00145/) (2022)
- [Reducing overestimating and underestimating volatility via the augmented blending-ARCH model](https://mlsanity.com/paper/2203.12456/) (2022)
- [Exploring Classic Quantitative Strategies](https://mlsanity.com/paper/2202.11309/) (2022)

## Built on by (most cited first, 2 of 2)

- [Gradient Descent, Stochastic Optimization, and Other Tales](https://mlsanity.com/paper/2205.00832/) (2022)
- [A note on VIX for postprocessing quantitative strategies](https://mlsanity.com/paper/2207.04887/) (2022)

## Abstract (arXiv)

It is well known that we need to choose the hyper-parameters in Momentum, AdaGrad, AdaDelta, and other alternative stochastic optimizers. While in many cases, the hyper-parameters are tuned tediously based on experience becoming more of an art than science. We present a novel per-dimension learning rate method for gradient descent called AdaSmooth. The method is insensitive to hyper-parameters thus it requires no manual tuning of the hyper-parameters like Momentum, AdaGrad, and AdaDelta methods. We show promising results compared to other methods on different convolutional neural networks, multi-layer perceptron, and alternative machine learning tasks. Empirical results demonstrate that AdaSmooth works well in practice and compares favorably to other stochastic optimization methods in neural networks.