# Inefficiency of K-FAC for Large Batch Size Training

arXiv: https://arxiv.org/abs/1903.06237 · HTML: https://mlsanity.com/paper/1903.06237/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2019.

## Built on by (most cited first, 4 of 4)

- [AdaFisher: Adaptive Second Order Optimization via Fisher Information](https://mlsanity.com/paper/2405.16397/) (2024)
- [Second-Order Neural ODE Optimizer](https://mlsanity.com/paper/2109.14158/) (2021)
- [A New Perspective for Understanding Generalization Gap of Deep Neural Networks Trained with Large Batch Sizes](https://mlsanity.com/paper/2210.12184/) (2022)
- [Non-Convex Optimization with Spectral Radius Regularization](https://mlsanity.com/paper/2102.11210/) (2021)

## Abstract (arXiv)

In stochastic optimization, using large batch sizes during training can leverage parallel resources to produce faster wall-clock training times per training epoch. However, for both training loss and testing error, recent results analyzing large batch Stochastic Gradient Descent (SGD) have found sharp diminishing returns, beyond a certain critical batch size. In the hopes of addressing this, it has been suggested that the Kronecker-Factored Approximate Curvature (mboxK-FAC) method allows for greater scalability to large batch sizes, for non-convex machine learning problems such as neural network optimization, as well as greater robustness to variation in model hyperparameters. Here, we perform a detailed empirical analysis of large batch size training %of these two hypotheses, for both mboxK-FAC and SGD, evaluating performance in terms of both wall-clock time and aggregate computational cost. Our main results are twofold: first, we find that both mboxK-FAC and SGD doesn't have ideal scalability behavior beyond a certain batch size, and that mboxK-FAC does not exhibit improved large-batch scalability behavior, as compared to SGD; and second, we find that mboxK-FAC, in addition to requiring more hyperparameters to tune, suffers from similar hyperparameter sensitivity behavior as does SGD. We discuss extensive results using ResNet and AlexNet on mboxCIFAR-10 and SVHN, respectively, as well as more general implications of our findings.