mlsanityData to 7 Oct 2026

Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam

arXiv 1806.04854 ↗First posted 13 Jun 2018stat.ML cs.AI cs.LGcited by 131

Not on the map. Why?

Abstract (arXiv)

Uncertainty computation in deep learning is essential to design robust and reliable systems. Variational inference (VI) is a promising approach for such computation, but requires more effort to implement and execute compared to maximum-likelihood methods. In this paper, we propose new natural-gradient algorithms to reduce such efforts for Gaussian mean-field VI. Our algorithms can be implemented within the Adam optimizer by perturbing the network weights during gradient evaluations, and uncertainty estimates can be cheaply obtained by using the vector that adapts the learning rate. This requires lower memory, computation, and implementation effort than existing VI methods, while obtaining uncertainty estimates of comparable quality. Our empirical results confirm this and further suggest that the weight-perturbation in our algorithm could be useful for exploration in reinforcement learning and stochastic optimization.

Key facts. Cited by 131 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 24; 2022: 23; 2023: 34; 2024: 26; 2025: 15; 2026: 8.

131 papers.

Sort:

By area:

  1. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  2. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  3. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  4. 2021The Bayesian Learning Rulecited by 95
  5. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  6. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  7. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  8. 2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 71
  9. 2024Variational Learning is Effective for Large Deep Networkscited by 62
  10. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  11. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  12. 2022SAM as an Optimal Relaxation of Bayescited by 40
38 more
  1. 2023Making Substitute Models More Bayesian Can Enhance Transferability of Adversarial Examplescited by 32
  2. 2023BayesDiff: Estimating Pixel-wise Uncertainty in Diffusion via Bayesian Inferencecited by 30
  3. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 29
  4. 2022Accelerated Linearized Laplace Approximation for Bayesian Deep Learningcited by 27
  5. 2021Bayesian Inference with Certifiable Adversarial Robustnesscited by 26
  6. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  7. 2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
  8. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  9. 2021A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applicationscited by 24
  10. 2022Approximate Bayesian Neural Operators: Uncertainty Quantification for Parametric PDEscited by 24
  11. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  12. 2022A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Modelscited by 22
  13. 2022Bayesian Learning for Neural Networks: an algorithmic surveycited by 22
  14. 2022Generalized Variational Inference in Function Spaces: Gaussian Measures meet Bayesian Deep Learningcited by 20
  15. 2023ASDL: A Unified Interface for Gradient Preconditioning in PyTorchcited by 20
  16. 2023Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shiftcited by 20
  17. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  18. 2021Dual Parameterization of Sparse Variational Gaussian Processescited by 19
  19. 2023Federated Learning as Variational Inference: A Scalable Expectation Propagation Approachcited by 17
  20. 2022Amortized Variational Inference: A Systematic Reviewcited by 16
  21. 2023Gradient Flows for Sampling: Mean-Field Models, Gaussian Approximations and Affine Invariancecited by 16
  22. 2023The Memory Perturbation Equation: Understanding Model's Sensitivity to Datacited by 16
  23. 2023Improving Neural Additive Models with Bayesian Principlescited by 15
  24. 2023Riemannian Laplace approximations for Bayesian neural networkscited by 14
  25. 2024Bayesian Online Natural Gradient (BONG)cited by 14
  26. 2021Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guaranteescited by 12
  27. 2024Sampling from Bayesian Neural Network Posteriors with Symmetric Minibatch Splitting Langevin Dynamicscited by 12
  28. 2023Towards Accelerated Model Training via Bayesian Data Selectioncited by 11
  29. 2024Linearization Turns Neural Operators into Function-Valued Gaussian Processescited by 11
  30. 2024Manifold Sampling for Differentiable Uncertainty in Radiance Fieldscited by 11
  31. 2023Flat Seeking Bayesian Neural Networkscited by 10
  32. 2023Promises and Pitfalls of the Linearized Laplace in Bayesian Optimizationcited by 10
  33. 2023Toward Adversarial Training on Contextualized Language Representationcited by 10
  34. 2024Training-Free Bayesianization for Low-Rank Adapters of Large Language Modelscited by 10
  35. 2022Wide Bayesian neural networks have a simple weight posterior: theory and accelerated samplingcited by 9
  36. 2023Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFACcited by 9
  37. 2021Certification of Iterative Predictions in Bayesian Neural Networkscited by 8
  38. 2023BNN-DP: Robustness Certification of Bayesian Neural Networks via Dynamic Programmingcited by 8

50 most cited of 131 All 131 (JSON)