mlsanityData to 7 Oct 2026

Theoretical characterisation of the Gauss-Newton conditioning in Neural Networks

arXiv 2411.02139 ↗First posted 4 Nov 2024cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

The Gauss-Newton (GN) matrix plays an important role in machine learning, most evident in its use as a preconditioning matrix for a wide family of popular adaptive methods to speed up optimization. Besides, it can also provide key insights into the optimization landscape of neural networks. In the context of deep neural networks, understanding the GN matrix involves studying the interaction between different weight matrices as well as the dependencies introduced by the data, thus rendering its analysis challenging. In this work, we take a first step towards theoretically characterizing the conditioning of the GN matrix in neural networks. We establish tight bounds on the condition number of the GN in deep linear networks of arbitrary depth and width, which we also extend to two-layer ReLU networks. We expand the analysis to further architectural components, such as residual connections and convolutional layers. Finally, we empirically validate the bounds and uncover valuable insights into the influence of the analyzed architectural components.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 51 of its 72 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

51 of 72 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  7. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  8. 2018Group Normalizationcited by 1,514
  9. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  11. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  12. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
38 more
  1. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  3. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  4. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 629
  5. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  6. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  7. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  8. 2014New insights and perspectives on the natural gradient methodcited by 435
  9. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  11. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  12. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  13. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  14. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  15. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  16. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  17. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  18. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  19. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  20. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  21. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  22. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  23. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  24. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  25. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  26. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  27. 2019The Difficulty of Training Sparse Neural Networkscited by 62
  28. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  29. 2020Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networkscited by 45
  30. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  31. 2019Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networkscited by 32
  32. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 21
  33. 2019The asymptotic spectrum of the Hessian of DNN throughout trainingcited by 19
  34. 2020Flatness is a False Friendcited by 19
  35. 2019Towards Understanding the Importance of Shortcut Connections in Residual Networkscited by 18
  36. 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
  37. 2022Phenomenology of Double Descent in Finite-Width Neural Networkscited by 11
  38. 2023The Hessian perspective into the Nature of Convolutional Neural Networkscited by 10

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2025Predictability Enables Parallelization of Nonlinear State Space Modelscited by 7
  2. 2025Spectral Collapse Drives Loss of Plasticity in Deep Continual Learningcited by 5
  3. 2024A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networkscited by 2
  4. 2026SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learningcited by 2
  5. 2026Depth, Not Data: An Analysis of Hessian Spectral Bifurcationcited by 1