# Angle based dynamic learning rate for gradient descent

arXiv: https://arxiv.org/abs/2304.10457 · HTML: https://mlsanity.com/paper/2304.10457/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 44 references that are in the map.
- First posted on arXiv 20 Apr 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients](https://mlsanity.com/paper/2010.07468/) (2020)
- [diffGrad: An Optimization Method for Convolutional Neural Networks](https://mlsanity.com/paper/1909.11015/) (2019)
- [Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate](https://mlsanity.com/paper/1805.07557/) (2018)
- [AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks](https://mlsanity.com/paper/2105.10190/) (2021)
- [Effects of Spectral Normalization in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2212.05331/) (2022)
- [Review of Extreme Multilabel Classification](https://mlsanity.com/paper/2302.05971/) (2023)

## Built on by (most cited first, 2 of 2)

- [SignMuon: Communication-Efficient Distributed Muon Optimization](https://mlsanity.com/paper/2605.16311/) (2026)
- [A Gauss-Newton Approach for Min-Max Optimization in Generative Adversarial Networks](https://mlsanity.com/paper/2404.07172/) (2024)

## Abstract (arXiv)

In our work, we propose a novel yet simple approach to obtain an adaptive learning rate for gradient-based descent methods on classification tasks. Instead of the traditional approach of selecting adaptive learning rates via the decayed expectation of gradient-based terms, we use the angle between the current gradient and the new gradient: this new gradient is computed from the direction orthogonal to the current gradient, which further helps us in determining a better adaptive learning rate based on angle history, thereby, leading to relatively better accuracy compared to the existing state-of-the-art optimizers. On a wide variety of benchmark datasets with prominent image classification architectures such as ResNet, DenseNet, EfficientNet, and VGG, we find that our method leads to the highest accuracy in most of the datasets. Moreover, we prove that our method is convergent.