Improving Levenberg-Marquardt Algorithm for Neural Networks
Not on the map. Why?
Abstract (arXiv)
We explore the usage of the Levenberg-Marquardt (LM) algorithm for regression (non-linear least squares) and classification (generalized Gauss-Newton methods) tasks in neural networks. We compare the performance of the LM method with other popular first-order algorithms such as SGD and Adam, as well as other second-order algorithms such as L-BFGS , Hessian-Free and KFAC. We further speed up the LM method by using adaptive momentum, learning rate line search, and uphill step acceptance.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 9 of its 15 references that are in the map. First posted on arXiv 17 Dec 2022.
Builds on
9 of 15 references.
References · 9
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
- 2022Adaptive Second Order Coresets for Data-efficient Machine Learningcited by 60
- 2015Preconditioned Stochastic Gradient Descentcited by 50
- 2019Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networkscited by 33
- 2018Preconditioner on Matrix Lie Group for SGDcited by 16
- 2022Black Box Lie Group Preconditioners for SGDcited by 9
- 2018Online Second Order Methods for Non-Convex Stochastic Optimizationscited by 2
- 2022Causal Structural Hypothesis Testing and Data Generation Modelscited by 1
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.