mlsanityData to 14 Oct 2026

Rethinking Gauss-Newton for learning over-parameterized models

arXiv 2302.02904 ↗First posted 6 Feb 2023cs.LG math.OCcited by 6

Not on the map. Why?

Abstract (arXiv)

This work studies the global convergence and implicit bias of Gauss Newton's (GN) when optimizing over-parameterized one-hidden layer networks in the mean-field regime. We first establish a global convergence result for GN in the continuous-time limit exhibiting a faster convergence rate compared to GD due to improved conditioning. We then perform an empirical study on a synthetic regression task to investigate the implicit bias of GN's method. While GN is consistently faster than GD in finding a global optimum, the learned model generalizes well on test data when starting from random initial weights with a small variance and using a small step size to slow down convergence. Specifically, our study shows that such a setting results in a hidden learning phenomenon, where the dynamics are able to recover features with good generalization properties despite the model having sub-optimal training and test performances due to an under-optimized linear layer. This study exhibits a trade-off between the convergence speed of GN and the generalization ability of the learned solution.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 36 of its 56 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

36 of 56 references.

References · 36

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  3. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  5. 2018On Lazy Training in Differentiable Programmingcited by 675
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  7. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  9. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  10. 2014New insights and perspectives on the natural gradient methodcited by 443
  11. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  12. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
24 more
  1. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  2. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  3. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 142
  4. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  5. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  6. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  7. 2018Does data interpolation contradict statistical optimality?cited by 95
  8. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  9. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  10. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
  11. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  12. 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
  13. 2019Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problemscited by 40
  14. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 39
  15. 2019Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networkscited by 33
  16. 2022Non-Convex Bilevel Games with Critical Point Selection Mapscited by 25
  17. 2020Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networkscited by 22
  18. 2021Regularized Newton Method with Global O(1/k2) Convergencecited by 22
  19. 2021Global Convergence of Three-layer Neural Networks in the Mean Field Regimecited by 16
  20. 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 13
  21. 2019Kernelized Wasserstein Natural Gradientcited by 12
  22. 2021Depth Without the Magic: Inductive Bias of Natural Gradient Descentcited by 5
  23. 2023Convergence beyond the over-parameterized regime using Rayleigh quotientscited by 5
  24. 2021Flexible Modification of Gauss-Newton Method and Its Stochastic Extensioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 1.

6 papers.

  1. 2024Exact Gauss-Newton Optimization for Training Deep Neural Networkscited by 10
  2. 2024Regularized Gauss-Newton for Optimizing Overparameterized Neural Networkscited by 3
  3. 2024A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networkscited by 3
  4. 2025Gradient-Normalized Smoothness for Optimization with Approximate Hessianscited by 3
  5. 2025Non-Asymptotic Optimization and Generalization Bounds for Stochastic Gauss-Newton in Overparameterized Modelscited by 0
  6. 2026Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limitcited by 0