# 2nd-order Updates with 1st-order Complexity

arXiv: https://arxiv.org/abs/2105.11439 · HTML: https://mlsanity.com/paper/2105.11439/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 50 references that are in the map.
- First posted on arXiv 24 May 2021.

## Builds on (most cited first, 3 of 3)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)

## Abstract (arXiv)

It has long been a goal to efficiently compute and use second order information on a function (f) to assist in numerical approximations. Here it is shown how, using only basic physics and a numerical approximation, such information can be accurately obtained at a cost of cal O(N) complexity, where N is the dimensionality of the parameter space of f. In this paper, an algorithm (em VA-Flow) is developed to exploit this second order information, and pseudocode is presented. It is applied to two classes of problems, that of inverse kinematics (IK) and gradient descent (GD). In the IK application, the algorithm is fast and robust, and is shown to lead to smooth behavior even near singularities. For GD the algorithm also works very well, provided the cost function is locally well-described by a polynomial.