mlsanityData to 14 Oct 2026

Practical Gauss-Newton Optimisation for Deep Learning

arXiv 1706.03662 ↗First posted 12 Jun 2017stat.MLcited by 142

Not on the map. Why?

Abstract (arXiv)

We present an efficient block-diagonal ap- proximation to the Gauss-Newton matrix for feedforward neural networks. Our result- ing algorithm is competitive against state- of-the-art first order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for which hyperparameter tuning of the optimisation parameters is often a labo- rious process, our approach can provide good performance even when used with default set- tings. A side result of our work is that for piecewise linear transfer functions, the net- work objective function can have no differ- entiable local maxima, which may partially explain why such transfer functions facilitate effective optimisation.

Key facts. Cited by 142 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 31; 2022: 18; 2023: 24; 2024: 28; 2025: 19; 2026: 22.

142 papers.

Sort:

By area:

  1. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
  2. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  3. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  4. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  5. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  6. 2021Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networkscited by 139
  7. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 106
  8. 2021Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Timecited by 62
  9. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  10. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  11. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 48
  12. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
38 more
  1. 2022Enhancing Adversarial Training with Second-Order Statistics of Weightscited by 42
  2. 2023The LLM Surgeoncited by 38
  3. 2024Influence Functions for Scalable Data Attribution in Diffusion Modelscited by 33
  4. 2021Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networkscited by 32
  5. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 30
  6. 2021Credit Assignment in Neural Networks through Deep Feedback Controlcited by 29
  7. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  8. 2024A Comprehensive Survey on Evidential Deep Learning and Its Applicationscited by 25
  9. 2021Tensor Normal Training for Deep Learning Modelscited by 24
  10. 2024EDA-DM: Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Modelscited by 23
  11. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  12. 2023FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregationcited by 22
  13. 2023ASDL: A Unified Interface for Gradient Preconditioning in PyTorchcited by 21
  14. 2024Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networkscited by 21
  15. 2023Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategiescited by 20
  16. 2023Fixing Overconfidence in Dynamic Neural Networkscited by 19
  17. 2023Variational Linearized Laplace Approximation for Bayesian Deep Learningcited by 18
  18. 2025Probabilistic neural operators for functional uncertainty quantificationcited by 18
  19. 2022Adversarial Patch Attacks and Defences in Vision-Based Tasks: A Surveycited by 17
  20. 2021ViViT: Curvature access through the generalized Gauss-Newton's low-rank structurecited by 16
  21. 2021Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrumscited by 15
  22. 2023Riemannian Laplace approximations for Bayesian neural networkscited by 15
  23. 2021Does the Data Induce Capacity Control in Deep Learning?cited by 14
  24. 2024BoA: Attention-aware Post-training Quantization without Backpropagationcited by 12
  25. 2021Second-Order Neural ODE Optimizercited by 11
  26. 2021Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problemscited by 11
  27. 2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 11
  28. 2021Minibatch and Momentum Model-based Methods for Stochastic Weakly Convex Optimizationcited by 10
  29. 2021Large-Scale Deep Learning Optimizations: A Comprehensive Surveycited by 10
  30. 2022Laplacian Autoencoders for Learning Stochastic Representationscited by 10
  31. 2022Rate-Distortion Optimized Post-Training Quantization for Learned Image Compressioncited by 10
  32. 2023Stochastic Marginal Likelihood Gradients using Neural Tangent Kernelscited by 10
  33. 2024Exact Gauss-Newton Optimization for Training Deep Neural Networkscited by 10
  34. 2024Sources of Uncertainty in 3D Scene Reconstructioncited by 10
  35. 2021Sparse Bayesian Deep Learning for Dynamic System Identificationcited by 9
  36. 2022Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decompositioncited by 9
  37. 2023Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFACcited by 9
  38. 2024Towards Next-Level Post-Training Quantization of Hyper-Scale Transformerscited by 9

50 most cited of 142 All 142 (JSON)