# SVD-Preconditioned Gradient Descent Method for Solving Nonlinear Least Squares Problems

arXiv: https://arxiv.org/abs/2602.09057 · HTML: https://mlsanity.com/paper/2602.09057/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 27 references that are in the map.
- First posted on arXiv 7 Feb 2026.

## Builds on (most cited first, 10 of 10)

- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Shampoo: Preconditioned Stochastic Tensor Optimization](https://mlsanity.com/paper/1802.09568/) (2018)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization](https://mlsanity.com/paper/1808.05671/) (2018)
- [Rank Diminishing in Deep Neural Networks](https://mlsanity.com/paper/2206.06072/) (2022)
- [Additive Schwarz Methods for Convex Optimization as Gradient Methods](https://mlsanity.com/paper/1912.03617/) (2019)
- [Accelerated Additive Schwarz Methods for Convex Optimization with Adaptive Restart](https://mlsanity.com/paper/2011.02695/) (2020)
- [Fisher-Informed Parameterwise Aggregation for Federated Learning with Heterogeneous Data](https://mlsanity.com/paper/2601.13608/) (2026)

## Abstract (arXiv)

This paper introduces a novel optimization algorithm designed for nonlinear least-squares problems. The method is derived by preconditioning the gradient descent direction using the Singular Value Decomposition (SVD) of the Jacobian. This SVD-based preconditioner is then integrated with the first- and second-moment adaptive learning rate mechanism of the Adam optimizer. We establish the local linear convergence of the proposed method under standard regularity assumptions and prove global convergence for a modified version of the algorithm under suitable conditions. The effectiveness of the approach is demonstrated experimentally across a range of tasks, including function approximation, partial differential equation (PDE) solving, and image classification on the CIFAR-10 dataset. Results show that the proposed method consistently outperforms standard Adam, achieving faster convergence and lower error in both regression and classification settings.