# The Lie-Group Bayesian Learning Rule

arXiv: https://arxiv.org/abs/2303.04397 · HTML: https://mlsanity.com/paper/2303.04397/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [Practical Deep Learning with Bayesian Principles](https://mlsanity.com/paper/1906.02506/) (2019)
- [Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam](https://mlsanity.com/paper/1806.04854/) (2018)
- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [Implicit Reparameterization Gradients](https://mlsanity.com/paper/1805.08498/) (2018)
- [Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models](https://mlsanity.com/paper/1703.04265/) (2017)
- [Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations](https://mlsanity.com/paper/1906.02914/) (2019)
- [Anticorrelated Noise Injection for Improved Generalization](https://mlsanity.com/paper/2202.02831/) (2022)
- [SAM as an Optimal Relaxation of Bayes](https://mlsanity.com/paper/2210.01620/) (2022)
- [Exponentiated Gradient Meets Gradient Descent](https://mlsanity.com/paper/1902.01903/) (2019)
- [Handling the Positive-Definite Constraint in the Bayesian Learning Rule](https://mlsanity.com/paper/2002.10060/) (2020)
- [The Generalized Reparameterization Gradient](https://mlsanity.com/paper/1610.02287/) (2016)
- [Disentanglement with Biological Constraints: A Theory of Functional Cell Types](https://mlsanity.com/paper/2210.01768/) (2022)
- [Training Binary Neural Networks using the Bayesian Learning Rule](https://mlsanity.com/paper/2002.10778/) (2020)
- [Tractable structured natural gradient descent using local parameterizations](https://mlsanity.com/paper/2102.07405/) (2021)
- [Learning compositional functions via multiplicative weight updates](https://mlsanity.com/paper/2006.14560/) (2020)
- [Stein's Lemma for the Reparameterization Trick with Exponential Family Mixtures](https://mlsanity.com/paper/1910.13398/) (2019)

## Built on by (most cited first, 3 of 3)

- [Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networks](https://mlsanity.com/paper/2506.17768/) (2025)
- [Information Geometry of Variational Bayes](https://mlsanity.com/paper/2509.15641/) (2025)
- [Variational Bayes Made Easy](https://mlsanity.com/paper/2304.14251/) (2023)

## Abstract (arXiv)

The Bayesian Learning Rule provides a framework for generic algorithm design but can be difficult to use for three reasons. First, it requires a specific parameterization of exponential family. Second, it uses gradients which can be difficult to compute. Third, its update may not always stay on the manifold. We address these difficulties by proposing an extension based on Lie-groups where posteriors are parametrized through transformations of an arbitrary base distribution and updated via the group's exponential map. This simplifies all three difficulties for many cases, providing flexible parametrizations through group's action, simple gradient computation through reparameterization, and updates that always stay on the manifold. We use the new learning rule to derive a new algorithm for deep learning with desirable biologically-plausible attributes to learn sparse features. Our work opens a new frontier for the design of new algorithms by exploiting Lie-group structures.