# Scalable and Practical Natural Gradient for Large-Scale Deep Learning

arXiv: https://arxiv.org/abs/2002.06015 · HTML: https://mlsanity.com/paper/2002.06015/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2020.

## Built on by (most cited first, 15 of 15)

- [Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations](https://mlsanity.com/paper/2202.10638/) (2022)
- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices](https://mlsanity.com/paper/2211.14133/) (2022)
- [ASDL: A Unified Interface for Gradient Preconditioning in PyTorch](https://mlsanity.com/paper/2305.04684/) (2023)
- [Natural Gradient Methods: Perspectives, Efficient-Scalable Approximations, and Analysis](https://mlsanity.com/paper/2303.05473/) (2023)
- [Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning](https://mlsanity.com/paper/2206.15143/) (2022)
- [Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks](https://mlsanity.com/paper/2107.06533/) (2021)
- [Beyond the Mean: Fisher-Orthogonal Projection for Natural Gradient Descent in Large Batch Training](https://mlsanity.com/paper/2508.13898/) (2025)
- [Randomized K-FACs: Speeding up K-FAC with Randomized Numerical Linear Algebra](https://mlsanity.com/paper/2206.15397/) (2022)
- [Brand New K-FACs: Speeding up K-FAC with Online Decomposition Updates](https://mlsanity.com/paper/2210.08494/) (2022)
- [Uncovering Model Processing Strategies with Non-Negative Per-Example Fisher Factorization](https://mlsanity.com/paper/2310.04649/) (2023)
- [Rethinking Exponential Averaging of the Fisher](https://mlsanity.com/paper/2204.04718/) (2022)
- [Eva: A General Vectorized Approximation Framework for Second-order Optimization](https://mlsanity.com/paper/2308.02123/) (2023)
- [Jorge: Approximate Preconditioning for GPU-efficient Second-order Optimization](https://mlsanity.com/paper/2310.12298/) (2023)
- [Second-Order, First-Class: A Composable Stack for Curvature-Aware Training](https://mlsanity.com/paper/2603.25976/) (2026)

## Abstract (arXiv)

Large-scale distributed training of deep neural networks results in models with worse generalization performance as a result of the increase in the effective mini-batch size. Previous approaches attempt to address this problem by varying the learning rate and batch size over epochs and layers, or ad hoc modifications of batch normalization. We propose Scalable and Practical Natural Gradient Descent (SP-NGD), a principled approach for training models that allows them to attain similar generalization performance to models trained with first-order optimization methods, but with accelerated convergence. Furthermore, SP-NGD scales to large mini-batch sizes with a negligible computational overhead as compared to first-order methods. We evaluated SP-NGD on a benchmark task where highly optimized first-order methods are available as references: training a ResNet-50 model for image classification on ImageNet. We demonstrate convergence to a top-1 validation accuracy of 75.4% in 5.5 minutes using a mini-batch size of 32,768 with 1,024 GPUs, as well as an accuracy of 74.9% with an extremely large mini-batch size of 131,072 in 873 steps of SP-NGD.