# Eigenvalue Corrected Noisy Natural Gradient

arXiv: https://arxiv.org/abs/1811.12565 · HTML: https://mlsanity.com/paper/1811.12565/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2018.

## Built on by (most cited first, 7 of 7)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks](https://mlsanity.com/paper/2107.01739/) (2021)
- [Distributional Training Data Attribution: What do Influence Functions Sample?](https://mlsanity.com/paper/2506.12965/) (2025)
- [SOAP-Bubbles: Structured Weight Uncertainty for Neural Networks](https://mlsanity.com/paper/2606.23357/) (2026)
- [Laplace Sample Information: Data Informativeness Through a Bayesian Lens](https://mlsanity.com/paper/2505.15303/) (2025)
- [Tempering the Bayes Filter towards Improved Model-Based Estimation](https://mlsanity.com/paper/2512.02823/) (2025)
- [Instance-Level Post Hoc Uncertainty Quantification in Object Detection](https://mlsanity.com/paper/2606.04656/) (2026)

## Abstract (arXiv)

Variational Bayesian neural networks combine the flexibility of deep learning with Bayesian uncertainty estimation. However, inference procedures for flexible variational posteriors are computationally expensive. A recently proposed method, noisy natural gradient, is a surprisingly simple method to fit expressive posteriors by adding weight noise to regular natural gradient updates. Noisy K-FAC is an instance of noisy natural gradient that fits a matrix-variate Gaussian posterior with minor changes to ordinary K-FAC. Nevertheless, a matrix-variate Gaussian posterior does not capture an accurate diagonal variance. In this work, we extend on noisy K-FAC to obtain a more flexible posterior distribution called eigenvalue corrected matrix-variate Gaussian. The proposed method computes the full diagonal re-scaling factor in Kronecker-factored eigenbasis. Empirically, our approach consistently outperforms existing algorithms (e.g., noisy K-FAC) on regression and classification tasks.