# Improving Levenberg-Marquardt Algorithm for Neural Networks

arXiv: https://arxiv.org/abs/2212.08769 · HTML: https://mlsanity.com/paper/2212.08769/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 15 references that are in the map.
- First posted on arXiv 17 Dec 2022.

## Builds on (most cited first, 9 of 9)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Adaptive Second Order Coresets for Data-efficient Machine Learning](https://mlsanity.com/paper/2207.13887/) (2022)
- [Preconditioned Stochastic Gradient Descent](https://mlsanity.com/paper/1512.04202/) (2015)
- [Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networks](https://mlsanity.com/paper/1906.02353/) (2019)
- [Preconditioner on Matrix Lie Group for SGD](https://mlsanity.com/paper/1809.10232/) (2018)
- [Black Box Lie Group Preconditioners for SGD](https://mlsanity.com/paper/2211.04422/) (2022)
- [Online Second Order Methods for Non-Convex Stochastic Optimizations](https://mlsanity.com/paper/1803.09383/) (2018)
- [Causal Structural Hypothesis Testing and Data Generation Models](https://mlsanity.com/paper/2210.11275/) (2022)

## Built on by (most cited first, 2 of 2)

- [Exact Gauss-Newton Optimization for Training Deep Neural Networks](https://mlsanity.com/paper/2405.14402/) (2024)
- [A pullback-corrected scalar auxiliary variable optimizer with momentum and adaptive mobility](https://mlsanity.com/paper/2609.13569/) (2026)

## Abstract (arXiv)

We explore the usage of the Levenberg-Marquardt (LM) algorithm for regression (non-linear least squares) and classification (generalized Gauss-Newton methods) tasks in neural networks. We compare the performance of the LM method with other popular first-order algorithms such as SGD and Adam, as well as other second-order algorithms such as L-BFGS , Hessian-Free and KFAC. We further speed up the LM method by using adaptive momentum, learning rate line search, and uphill step acceptance.