# Nonlinear Least Squares for Large-Scale Machine Learning using Stochastic Jacobian Estimates

arXiv: https://arxiv.org/abs/2107.05598 · HTML: https://mlsanity.com/paper/2107.05598/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 14 references that are in the map.
- First posted on arXiv 12 Jul 2021.

## Builds on (most cited first, 5 of 5)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study](https://mlsanity.com/paper/1708.07827/) (2017)
- [A stochastic Levenberg-Marquardt method using random models with complexity results](https://mlsanity.com/paper/1807.02176/) (2018)

## Built on by (most cited first, 4 of 4)

- [Exact Gauss-Newton Optimization for Training Deep Neural Networks](https://mlsanity.com/paper/2405.14402/) (2024)
- [Incremental Gauss-Newton Descent for Machine Learning](https://mlsanity.com/paper/2408.05560/) (2024)
- [A Nonmonotone Gradient-Based Algorithm for Symmetric Nonnegative Matrix Factorization and Graph Clustering](https://mlsanity.com/paper/2606.02887/) (2026)
- [Constrained Variable Projection for Structured Problems](https://mlsanity.com/paper/2606.23939/) (2026)

## Abstract (arXiv)

For large nonlinear least squares loss functions in machine learning we exploit the property that the number of model parameters typically exceeds the data in one batch. This implies a low-rank structure in the Hessian of the loss, which enables effective means to compute search directions. Using this property, we develop two algorithms that estimate Jacobian matrices and perform well when compared to state-of-the-art methods.