# Gradient descent revisited via an adaptive online learning rate

arXiv: https://arxiv.org/abs/1801.09136 · HTML: https://mlsanity.com/paper/1801.09136/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jan 2018.

## Built on by (most cited first, 1 of 1)

- [Optimizing Convergence for Iterative Learning of ARIMA for Stationary Time Series](https://mlsanity.com/paper/2101.10037/) (2021)

## Abstract (arXiv)

Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the gradient descent algorithm in the which the learning rate is not fixed. Instead, we learn the learning rate itself, either by another gradient descent (first-order method), or by Newton's method (second-order). This way, gradient descent for any machine learning algorithm can be optimized.