mlsanityData to 7 Oct 2026

Noisy Natural Gradient as Variational Inference

arXiv 1712.02390 ↗First posted 6 Dec 2017cs.LG stat.MLcited by 94

Not on the map. Why?

Abstract (arXiv)

Variational Bayesian neural nets combine the flexibility of deep learning with Bayesian uncertainty estimation. Unfortunately, there is a tradeoff between cheap but simple variational families (e.g.~fully factorized) or expensive and complicated inference procedures. We show that natural gradient ascent with adaptive weight noise implicitly fits a variational posterior to maximize the evidence lower bound (ELBO). This insight allows us to train full-covariance, fully factorized, or matrix-variate Gaussian variational posteriors using noisy versions of natural gradient, Adam, and K-FAC, respectively, making it possible to scale up to modern-size ConvNets. On standard regression benchmarks, our noisy K-FAC algorithm makes better predictions and matches Hamiltonian Monte Carlo's predictive variances better than existing methods. Its improved uncertainty estimates lead to more efficient exploration in active learning, and intrinsic motivation for reinforcement learning.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 25; 2022: 16; 2023: 26; 2024: 13; 2025: 8; 2026: 5.

94 papers.

Sort:

By area:

  1. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  2. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  3. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  4. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  5. 2021The Bayesian Learning Rulecited by 95
  6. 2022Variational inference via Wasserstein gradient flowscited by 84
  7. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
  8. 2021On the Practicality of Deterministic Epistemic Uncertaintycited by 62
  9. 2024Variational Learning is Effective for Large Deep Networkscited by 62
  10. 2023Prediction-Oriented Bayesian Active Learningcited by 60
  11. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  12. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
38 more
  1. 2021Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equationscited by 38
  2. 2021Data augmentation in Bayesian neural networks and the cold posterior effectcited by 36
  3. 2021Differentiable Annealed Importance Sampling and the Perils of Gradient Noisecited by 36
  4. 2023Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Spacecited by 36
  5. 2023Making Substitute Models More Bayesian Can Enhance Transferability of Adversarial Examplescited by 32
  6. 2023BayesDiff: Estimating Pixel-wise Uncertainty in Diffusion via Bayesian Inferencecited by 30
  7. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 29
  8. 2022Accelerated Linearized Laplace Approximation for Bayesian Deep Learningcited by 27
  9. 2021Bayesian Inference with Certifiable Adversarial Robustnesscited by 26
  10. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  11. 2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
  12. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  13. 2021A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applicationscited by 24
  14. 2022Approximate Bayesian Neural Operators: Uncertainty Quantification for Parametric PDEscited by 24
  15. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  16. 2022Masking Adversarial Damage: Finding Adversarial Saliency for Robust and Sparse Networkcited by 21
  17. 2021KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networkscited by 20
  18. 2023ASDL: A Unified Interface for Gradient Preconditioning in PyTorchcited by 20
  19. 2023Federated Learning as Variational Inference: A Scalable Expectation Propagation Approachcited by 17
  20. 2021Sparse Uncertainty Representation in Deep Learning with Inducing Weightscited by 16
  21. 2021Being a Bit Frequentist Improves Bayesian Neural Networkscited by 15
  22. 2023NPCL: Neural Processes for Uncertainty-Aware Continual Learningcited by 14
  23. 2024Scalable Bayesian Learning with posteriorscited by 13
  24. 2022On the Robustness of Bayesian Neural Networks to Adversarial Attackscited by 12
  25. 2023Towards Accelerated Model Training via Bayesian Data Selectioncited by 11
  26. 2024Linearization Turns Neural Operators into Function-Valued Gaussian Processescited by 11
  27. 2024Manifold Sampling for Differentiable Uncertainty in Radiance Fieldscited by 11
  28. 2021Model Selection for Bayesian Autoencoderscited by 10
  29. 2022Laplacian Autoencoders for Learning Stochastic Representationscited by 10
  30. 2023Flat Seeking Bayesian Neural Networkscited by 10
  31. 2025Foundations of Top-k Decoding For Language Modelscited by 10
  32. 2025Better Training Data Attribution via Better Inverse Hessian-Vector Productscited by 10
  33. 2022Hierarchically Structured Task-Agnostic Continual Learningcited by 9
  34. 2023Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFACcited by 9
  35. 2024EVCL: Elastic Variational Continual Learning with Weight Consolidationcited by 9
  36. 2021Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Drivingcited by 8
  37. 2023BNN-DP: Robustness Certification of Bayesian Neural Networks via Dynamic Programmingcited by 8
  38. 2024Understanding Stochastic Natural Gradient Variational Inferencecited by 8

50 most cited of 94 All 94 (JSON)