mlsanityData to 14 Oct 2026

Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning

arXiv 2206.15143 ↗First posted 30 Jun 2022cs.LG cs.DCcited by 7

Not on the map. Why?

Abstract (arXiv)

The second-order optimization methods, notably the D-KFAC (Distributed Kronecker Factored Approximate Curvature) algorithms, have gained traction on accelerating deep neural network (DNN) training on GPU clusters. However, existing D-KFAC algorithms require to compute and communicate a large volume of second-order information, i.e., Kronecker factors (KFs), before preconditioning gradients, resulting in large computation and communication overheads as well as a high memory footprint. In this paper, we propose DP-KFAC, a novel distributed preconditioning scheme that distributes the KF constructing tasks at different DNN layers to different workers. DP-KFAC not only retains the convergence property of the existing D-KFAC algorithms but also enables three benefits: reduced computation overhead in constructing KFs, no communication of KFs, and low memory footprint. Extensive experiments on a 64-GPU cluster show that DP-KFAC reduces the computation overhead by 1.55x-1.65x, the communication cost by 2.79x-3.15x, and the memory footprint by 1.14x-1.47x in each second-order update compared to the state-of-the-art D-KFAC methods.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 36 of its 63 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

36 of 63 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2016Densely Connected Convolutional Networkscited by 6,720
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  11. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  12. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
24 more
  1. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  2. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  3. 2014New insights and perspectives on the natural gradient methodcited by 443
  4. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  5. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  6. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  7. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  8. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  9. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 146
  10. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  11. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  12. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  13. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 57
  14. 2019On the interplay between noise and curvature and its effect on optimization and generalizationcited by 44
  15. 2018Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networkscited by 37
  16. 2020When Does Preconditioning Help or Hurt Generalization?cited by 31
  17. 2020Extrapolation for Large-batch Training in Deep Learningcited by 25
  18. 2020Convolutional Neural Network Training with Distributed K-FACcited by 22
  19. 2021KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networkscited by 20
  20. 2019On Convergence of Distributed Approximate Newton Methods: Globalization, Sharper Bounds and Beyondcited by 18
  21. 2020Scalable and Practical Natural Gradient for Large-Scale Deep Learningcited by 15
  22. 2020A Trace-restricted Kronecker-Factored Approximation to Natural Gradientcited by 11
  23. 2021TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversioncited by 7
  24. 2021Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Taskscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 0; 2025: 0; 2026: 2.

7 papers.

  1. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  2. 2023FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregationcited by 22
  3. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 15
  4. 2023Eva: A General Vectorized Approximation Framework for Second-order Optimizationcited by 1
  5. 2026Efficient Bilevel Optimization with KFAC-Based Hypergradientscited by 1
  6. 2026Layerwise LQR for Geometry-Aware Optimization of Deep Networkscited by 1
  7. 2023Dual Gauss-Newton Directions for Deep Learningcited by 0