# First-order and second-order variants of the gradient descent in a unified framework

arXiv: https://arxiv.org/abs/1810.08102 · HTML: https://mlsanity.com/paper/1810.08102/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Oct 2018.

## Built on by (most cited first, 3 of 3)

- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [Utilizing Evolution Strategies to Train Transformers in Reinforcement Learning](https://mlsanity.com/paper/2501.13883/) (2025)
- [Utilizing Novelty-based Evolution Strategies to Train Transformers in Reinforcement Learning](https://mlsanity.com/paper/2502.06301/) (2025)

## Abstract (arXiv)

In this paper, we provide an overview of first-order and second-order variants of the gradient descent method that are commonly used in machine learning. We propose a general framework in which 6 of these variants can be interpreted as different instances of the same approach. They are the vanilla gradient descent, the classical and generalized Gauss-Newton methods, the natural gradient descent method, the gradient covariance matrix approach, and Newton's method. Besides interpreting these methods within a single framework, we explain their specificities and show under which conditions some of them coincide.