mlsanityData to 14 Oct 2026

On the Parameterization of Second-Order Optimization Effective Towards the Infinite Width

arXiv 2312.12226 ↗First posted 19 Dec 2023cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Second-order optimization has been developed to accelerate the training of deep neural networks and it is being applied to increasingly larger-scale models. In this study, towards training on further larger scales, we identify a specific parameterization for second-order optimization that promotes feature learning in a stable manner even if the network width increases significantly. Inspired by a maximal update parameterization, we consider a one-step update of the gradient and reveal the appropriate scales of hyperparameters including random initialization, learning rates, and damping terms. Our approach covers two major second-order optimization algorithms, K-FAC and Shampoo, and we demonstrate that our parameterization achieves higher generalization performance in feature learning. In particular, it enables us to transfer the hyperparameters across models with different widths.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 44 of its 56 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

44 of 56 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  6. 2018On Lazy Training in Differentiable Programmingcited by 675
  7. 2017Deep Neural Networks as Gaussian Processescited by 622
  8. 2014New insights and perspectives on the natural gradient methodcited by 443
  9. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  10. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  11. 2021Knowledge distillation: A good teacher is patient and consistentcited by 259
  12. 2016Deep Information Propagationcited by 195
32 more
  1. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  2. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  3. 2013Revisiting Natural Gradient for Deep Networkscited by 159
  4. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  5. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  6. 2018Three Mechanisms of Weight Decay Regularizationcited by 113
  7. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  8. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 91
  9. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  10. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 67
  11. 2020Neural Kernels Without Tangentscited by 56
  12. 2015Preconditioned Stochastic Gradient Descentcited by 50
  13. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 41
  14. 2019Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problemscited by 40
  15. 2023Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limitcited by 38
  16. 2019Pathological spectra of the Fisher information metric and its variants in deep neural networkscited by 32
  17. 2020When Does Preconditioning Help or Hurt Generalization?cited by 31
  18. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 30
  19. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  20. 2022PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matricescited by 24
  21. 2020Convolutional Neural Network Training with Distributed K-FACcited by 22
  22. 2020Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networkscited by 22
  23. 2023Effective Theory of Transformers at Initializationcited by 21
  24. 2023ASDL: A Unified Interface for Gradient Preconditioning in PyTorchcited by 21
  25. 2021KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networkscited by 20
  26. 2019The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Studycited by 16
  27. 2021LocoProp: Enhancing BackProp via Local Loss Optimizationcited by 15
  28. 2020A Trace-restricted Kronecker-Factored Approximation to Natural Gradientcited by 11
  29. 2020Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Trainingcited by 11
  30. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  31. 2022Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioningcited by 7
  32. 2021Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Taskscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 7.

15 papers.

Sort:

  1. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  2. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 21
  3. 2025FedMuon: Federated Learning with Bias-corrected LMO-based Optimizationcited by 8
  4. 2025Pre-Training LLMs on a budget: A comparison of three optimizerscited by 5
  5. 2024Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagationcited by 4
  6. 2025Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stabilitycited by 2
  7. 2026FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoocited by 2
  8. 2026Statistical Properties of Training & Generalizationcited by 2
  9. 2025Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operatorscited by 1
  10. 2026Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Losscited by 1
  11. 2025How Does Preconditioning Guide Feature Learning in Deep Neural Networks?cited by 0
  12. 2026Towards a Principled Muon under μmathsfP: Ensuring Spectral Conditions throughout Trainingcited by 0
3 more
  1. 2026Extending μP: Spectral Conditions for Feature Learning Across Optimizerscited by 0
  2. 2026Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limitcited by 0
  3. 2026Unlocking Feature Learning in Gated Delta Networks at Scalecited by 0