mlsanityData to 7 Oct 2026

Analysis and Comparison of Two-Level KFAC Methods for Training Deep Neural Networks

arXiv 2303.18083 ↗First posted 31 Mar 2023cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

As a second-order method, the Natural Gradient Descent (NGD) has the ability to accelerate training of neural networks. However, due to the prohibitive computational and memory costs of computing and inverting the Fisher Information Matrix (FIM), efficient approximations are necessary to make NGD scalable to Deep Neural Networks (DNNs). Many such approximations have been attempted. The most sophisticated of these is KFAC, which approximates the FIM as a block-diagonal matrix, where each block corresponds to a layer of the neural network. By doing so, KFAC ignores the interactions between different layers. In this work, we investigate the interest of restoring some low-frequency interactions between the layers by means of two-level methods. Inspired from domain decomposition, several two-level corrections to KFAC using different coarse spaces are proposed and assessed. The obtained results show that incorporating the layer interactions in this fashion does not really improve the performance of KFAC. This suggests that it is safe to discard the off-diagonal blocks of the FIM, since the block-diagonal approach is sufficiently robust, accurate and economical in computation time.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 61 references that are in the map. First posted on arXiv 31 Mar 2023.

Builds on

25 of 61 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  7. 2017Convolutional Sequence to Sequence Learningcited by 550
  8. 2014New insights and perspectives on the natural gradient methodcited by 435
  9. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  10. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 143
  11. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  12. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
13 more
  1. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
  2. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  3. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  4. 2020Practical Quasi-Newton Methods for Training Deep Neural Networkscited by 67
  5. 2018Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networkscited by 37
  6. 2013Riemannian metrics for neural networks I: feedforward networkscited by 29
  7. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 29
  8. 2020Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networkscited by 22
  9. 2020A Trace-restricted Kronecker-Factored Approximation to Natural Gradientcited by 11
  10. 2015Very Deep Multilingual Convolutional Neural Networks for LVCSRcited by 10
  11. 2022Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decompositioncited by 9
  12. 2019Gradient Descent based Optimization Algorithms for Deep Learning Models Trainingcited by 6
  13. 2020Two-Level K-FAC Preconditioning for Deep Learningcited by 3

Built on by

Not cited yet.