mlsanityData to 7 Oct 2026

Regularized Gauss-Newton for Optimizing Overparameterized Neural Networks

arXiv 2404.14875 ↗First posted 23 Apr 2024cs.LG math.OCcited by 3

Not on the map. Why?

Abstract (arXiv)

The generalized Gauss-Newton (GGN) optimization method incorporates curvature estimates into its solution steps, and provides a good approximation to the Newton method for large-scale optimization problems. GGN has been found particularly interesting for practical training of deep neural networks, not only for its impressive convergence speed, but also for its close relation with neural tangent kernel regression, which is central to recent studies that aim to understand the optimization and generalization properties of neural networks. This work studies a GGN method for optimizing a two-layer neural network with explicit regularization. In particular, we consider a class of generalized self-concordant (GSC) functions that provide smooth approximations to commonly-used penalty terms in the objective function of the optimization problem. This approach provides an adaptive learning rate selection technique that requires little to no tuning for optimal performance. We study the convergence of the two-layer neural network, considered to be overparameterized, in the optimization loop of the resulting GGN method for a given scaling of the network parameters. Our numerical experiments highlight specific aspects of GSC regularization that help to improve generalization of the optimized neural network. The code to reproduce the experimental results is available at https://github.com/adeyemiadeoye/ggn-score-nn.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 39 of its 65 references that are in the map. First posted on arXiv 23 Apr 2024.

Builds on

39 of 65 references.

References · 39

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  6. 2018On Lazy Training in Differentiable Programmingcited by 671
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  9. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  11. 2014New insights and perspectives on the natural gradient methodcited by 435
  12. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
27 more
  1. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  2. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  3. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  4. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  5. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  6. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  7. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  8. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  9. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  10. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  11. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  12. 2017SGD Learns the Conjugate Kernel Class of the Networkcited by 54
  13. 2019Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problemscited by 40
  14. 2019RSN: Randomized Subspace Newtoncited by 34
  15. 2017Generalized Self-Concordant Functions: A Recipe for Newton-Type Methodscited by 33
  16. 2019Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networkscited by 32
  17. 2020Acceleration with a Ball Optimization Oraclecited by 31
  18. 2022Explicit Regularization in Overparametrized Models via Noise Injectioncited by 25
  19. 2018Global linear convergence of Newton's method without strong-convexity or Lipschitz gradientscited by 24
  20. 2020Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networkscited by 22
  21. 2019Kernelized Wasserstein Natural Gradientcited by 12
  22. 2021Training Recurrent Neural Networks by Sequential Least Squares and the Alternating Direction Method of Multiplierscited by 8
  23. 2020Explicit Regularization of Stochastic Gradient Methods through Dualitycited by 6
  24. 2023Rethinking Gauss-Newton for learning over-parameterized modelscited by 6
  25. 2021Depth Without the Magic: Inductive Bias of Natural Gradient Descentcited by 4
  26. 2021SCORE: Approximating Curvature Information under Self-Concordant Regularizationcited by 3
  27. 2023Self-concordant smoothing in proximal quasi-Newton algorithms for large-scale convex composite optimizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2024A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networkscited by 2
  2. 2025Noise-robust Contrastive Learning for Critical Transition Detection in Dynamical Systemscited by 0
  3. 2026Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limitcited by 0