# Modified Gauss-Newton Algorithms under Noise

arXiv: https://arxiv.org/abs/2305.10634 · HTML: https://mlsanity.com/paper/2305.10634/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 32 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 8 of 8)

- [Stochastic model-based minimization of weakly convex functions](https://mlsanity.com/paper/1803.06523/) (2018)
- [Stochastic Compositional Gradient Descent: Algorithms for Minimizing Compositions of Expected-Value Functions](https://mlsanity.com/paper/1411.3803/) (2014)
- [Efficiency of minimizing compositions of convex functions and smooth maps](https://mlsanity.com/paper/1605.00125/) (2016)
- [Torch-Struct: Deep Structured Prediction Library](https://mlsanity.com/paper/2002.00876/) (2020)
- [Stochastic Gauss-Newton Algorithms for Nonconvex Compositional Optimization](https://mlsanity.com/paper/2002.07290/) (2020)
- [A Smoother Way to Train Structured Prediction Models](https://mlsanity.com/paper/1902.03228/) (2019)
- [Stochastic Variance-Reduced Prox-Linear Algorithms for Nonconvex Composite Optimization](https://mlsanity.com/paper/2004.04357/) (2020)
- [Iterative Linearized Control: Stable Algorithms and Complexity Guarantees](https://mlsanity.com/paper/1908.07615/) (2019)

## Built on by (most cited first, 1 of 1)

- [Dual Gauss-Newton Directions for Deep Learning](https://mlsanity.com/paper/2308.08886/) (2023)

## Abstract (arXiv)

Gauss-Newton methods and their stochastic version have been widely used in machine learning and signal processing. Their nonsmooth counterparts, modified Gauss-Newton or prox-linear algorithms, can lead to contrasting outcomes when compared to gradient descent in large-scale statistical settings. We explore the contrasting performance of these two classes of algorithms in theory on a stylized statistical example, and experimentally on learning problems including structured prediction. In theory, we delineate the regime where the quadratic convergence of the modified Gauss-Newton method is active under statistical noise. In the experiments, we underline the versatility of stochastic (sub)-gradient descent to minimize nonsmooth composite objectives.