mlsanityData to 7 Oct 2026

The Bayesian Learning Rule

arXiv 2107.04562 ↗First posted 9 Jul 2021stat.ML cs.LGcited by 95

Not on the map. Why?

Abstract (arXiv)

We show that many machine-learning algorithms are specific instances of a single algorithm called the Bayesian learning rule. The rule, derived from Bayesian principles, yields a wide-range of algorithms from fields such as optimization, deep learning, and graphical models. This includes classical algorithms such as ridge regression, Newton's method, and Kalman filter, as well as modern deep-learning algorithms such as stochastic-gradient descent, RMSprop, and Dropout. The key idea in deriving such algorithms is to approximate the posterior using candidate distributions estimated by using natural gradients. Different candidate distributions result in different algorithms and further approximations to natural gradients give rise to variants of those algorithms. Our work not only unifies, generalizes, and improves existing algorithms, but also helps us design new ones.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 52 of its 153 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

52 of 153 references.

References · 52

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  5. 2016Variational Inference: A Review for Statisticianscited by 1,466
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  8. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  9. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  10. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  11. 2015Weight Uncertainty in Neural Networkscited by 554
  12. 2013Generating Sequences With Recurrent Neural Networkscited by 549
38 more
  1. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  2. 2014New insights and perspectives on the natural gradient methodcited by 435
  3. 2013Gaussian Processes for Big Datacited by 377
  4. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  5. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  6. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  7. 2017Advances in Variational Inferencecited by 236
  8. 2017Concrete Dropoutcited by 227
  9. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
  10. 2013A General Framework for Updating Belief Distributionscited by 172
  11. 2019Practical Deep Learning with Bayesian Principlescited by 158
  12. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  13. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  14. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  15. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  16. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  17. 2016Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriorscited by 82
  18. 2017Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Modelscited by 65
  19. 2018An Introduction to Probabilistic Programmingcited by 62
  20. 2013Relative Loss Bounds for On-line Density Estimation with the Exponential Family of Distributionscited by 59
  21. 2019Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimizationcited by 58
  22. 2013The Information Geometry of Mirror Descentcited by 53
  23. 2019Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximationscited by 51
  24. 2015Bayesian Dark Knowledgecited by 45
  25. 2018Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Modelscited by 38
  26. 2018Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Modelscited by 37
  27. 2015On Graduated Optimization for Stochastic Non-Convex Problemscited by 34
  28. 2018SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradientcited by 34
  29. 2018The Many Faces of Exponential Weights in Online Learningcited by 33
  30. 2017Online Natural Gradient as a Kalman Filtercited by 25
  31. 2020Training Binary Neural Networks using the Bayesian Learning Rulecited by 22
  32. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  33. 2020Fast Variational Learning in State-Space Gaussian Process Modelscited by 20
  34. 2019Stein's Lemma for the Reparameterization Trick with Exponential Family Mixturescited by 18
  35. 2020Homeomorphic-Invariance of EM: Non-Asymptotic Convergence in KL Divergence for Exponential Families via Mirror Descentcited by 18
  36. 2016Bayesian Computing with INLA: A Reviewcited by 14
  37. 2018Bayesian filtering unifies adaptive and non-adaptive neural network optimization methodscited by 9
  38. 2015Evolutionary Multimodal Optimization: A Short Surveycited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 12; 2023: 15; 2024: 12; 2025: 28; 2026: 23.

95 papers.

Sort:

By area:

  1. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  2. 2022Variational inference via Wasserstein gradient flowscited by 84
  3. 2024Variational Learning is Effective for Large Deep Networkscited by 62
  4. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  5. 2021Bayesian Learning via Neural Schr"odinger-F"ollmer Flowscited by 56
  6. 2023A Study of Bayesian Neural Network Surrogates for Bayesian Optimizationcited by 50
  7. 2022SAM as an Optimal Relaxation of Bayescited by 40
  8. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  9. 2024Outlier-robust Kalman Filtering through Generalised Bayescited by 21
  10. 2023Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shiftcited by 20
  11. 2023The Memory Perturbation Equation: Understanding Model's Sensitivity to Datacited by 16
  12. 2024Bayesian Online Natural Gradient (BONG)cited by 14
38 more
  1. 2021Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guaranteescited by 12
  2. 2022Bayesian Continual Learning via Spiking Neural Networkscited by 12
  3. 2021AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networkscited by 11
  4. 2023Memory-Based Dual Gaussian Processes for Sequential Learningcited by 11
  5. 2023The Interpolating Information Criterion for Overparameterized Modelscited by 11
  6. 2022Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processescited by 10
  7. 2025Uncertainty-Aware Decoding with Minimum Bayes Riskcited by 10
  8. 2023Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Reviewcited by 9
  9. 2023Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFACcited by 9
  10. 2025Bayesian continual learning and forgetting in neural networkscited by 8
  11. 2022Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problemscited by 7
  12. 2022Fantasizing with Dual GPs in Bayesian Optimization and Active Learningcited by 7
  13. 2024eXponential FAmily Dynamical Systems (XFADS): Large-scale nonlinear Gaussian state-space modelingcited by 7
  14. 2025Hellinger-Kantorovich Gradient Flows: Global Exponential Decay of Entropy Functionalscited by 7
  15. 2025Knowledge Adaptation as Posterior Correctioncited by 6
  16. 2022Robustness to corruption in pre-trained Bayesian neural networkscited by 5
  17. 2022Compressed Particle-Based Federated Bayesian Learning and Unlearningcited by 5
  18. 2024A unifying framework for generalised Bayesian online learning in non-stationary environmentscited by 5
  19. 2021A Survey of Learning Criteria Going Beyond the Usual Riskcited by 4
  20. 2023Probabilistic Reach-Avoid for Bayesian Neural Networkscited by 4
  21. 2023Implicit Maximum a Posteriori Filtering via Adaptive Optimizationcited by 4
  22. 2024Inclusive KL Gradient Flows: Otto-Wasserstein, Fisher-Rao-Gaussian, and Local-Estimator Dynamicscited by 4
  23. 2025Connecting Federated ADMM to Bayescited by 4
  24. 2025Martingale Posterior Neural Networks for Fast Sequential Decision Makingcited by 4
  25. 2025SING: SDE Inference via Natural Gradientscited by 4
  26. 2025Improved Stochastic Optimization of LogSumExpcited by 4
  27. 2025Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rulescited by 4
  28. 2022Quasi Black-Box Variational Inference with Natural Gradients for Bayesian Learningcited by 3
  29. 2023The Lie-Group Bayesian Learning Rulecited by 3
  30. 2025Federated ADMM from Bayesian Dualitycited by 3
  31. 2025Optimization Guarantees for Square-Root Natural-Gradient Variational Inferencecited by 3
  32. 2026Position: agentic AI orchestration should be Bayes-consistentcited by 3
  33. 2022AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networkscited by 2
  34. 2023Bayesian Inference on Binary Spiking Networks Leveraging Nanoscale Device Stochasticitycited by 2
  35. 2024Evaluating Bayesian deep learning for radio galaxy classificationcited by 2
  36. 2025Distilling Calibration via Conformalized Credal Inferencecited by 2
  37. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  38. 2025Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networkscited by 2

50 most cited of 95 All 95 (JSON)