# Handling the Positive-Definite Constraint in the Bayesian Learning Rule

arXiv: https://arxiv.org/abs/2002.10060 · HTML: https://mlsanity.com/paper/2002.10060/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2020.

## Built on by (most cited first, 30 of 31)

- [Variational Learning is Effective for Large Deep Networks](https://mlsanity.com/paper/2402.17641/) (2024)
- [Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Space](https://mlsanity.com/paper/2304.05398/) (2023)
- [Bayesian Inference with Certifiable Adversarial Robustness](https://mlsanity.com/paper/2102.05289/) (2021)
- [Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective](https://mlsanity.com/paper/2402.03496/) (2024)
- [Bayesian Learning for Neural Networks: an algorithmic survey](https://mlsanity.com/paper/2211.11865/) (2022)
- [Tractable structured natural gradient descent using local parameterizations](https://mlsanity.com/paper/2102.07405/) (2021)
- [A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Models](https://mlsanity.com/paper/2209.11533/) (2022)
- [Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift](https://mlsanity.com/paper/2306.12306/) (2023)
- [The Memory Perturbation Equation: Understanding Model's Sensitivity to Data](https://mlsanity.com/paper/2310.19273/) (2023)
- [Amortized In-Context Bayesian Posterior Estimation](https://mlsanity.com/paper/2502.06601/) (2025)
- [Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guarantees](https://mlsanity.com/paper/2111.01721/) (2021)
- [Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC](https://mlsanity.com/paper/2312.05705/) (2023)
- [Wasserstein Gradient Flows for Optimizing Gaussian Mixture Policies](https://mlsanity.com/paper/2305.10411/) (2023)
- [Understanding Stochastic Natural Gradient Variational Inference](https://mlsanity.com/paper/2406.01870/) (2024)
- [Structured second-order methods via natural gradient descent](https://mlsanity.com/paper/2107.10884/) (2021)
- [Physics-Informed Variational State-Space Gaussian Processes](https://mlsanity.com/paper/2409.13876/) (2024)
- [Connecting Federated ADMM to Bayes](https://mlsanity.com/paper/2501.17325/) (2025)
- [Quasi Black-Box Variational Inference with Natural Gradients for Bayesian Learning](https://mlsanity.com/paper/2205.11568/) (2022)
- [The Lie-Group Bayesian Learning Rule](https://mlsanity.com/paper/2303.04397/) (2023)
- [SOAP-Bubbles: Structured Weight Uncertainty for Neural Networks](https://mlsanity.com/paper/2606.23357/) (2026)
- [Variational Learning Finds Flatter Solutions at the Edge of Stability](https://mlsanity.com/paper/2506.12903/) (2025)
- [Information Geometry of Variational Bayes](https://mlsanity.com/paper/2509.15641/) (2025)
- [Stein Latent Optimization for Generative Adversarial Networks](https://mlsanity.com/paper/2106.05319/) (2021)
- [Manifold Gaussian Variational Bayes on the Precision Matrix](https://mlsanity.com/paper/2210.14598/) (2022)
- [Joint State Estimation and Noise Identification Based on Variational Optimization](https://mlsanity.com/paper/2312.09585/) (2023)
- [Convergence of projected stochastic natural gradient variational inference for various step size and sample or batch size schedules](https://mlsanity.com/paper/2604.00683/) (2026)
- [Parameter Exploration for RLVR via Variational Learning](https://mlsanity.com/paper/2608.09805/) (2026)
- [Latent Task-Specific Graph Network Simulators](https://mlsanity.com/paper/2311.05256/) (2023)
- [Spectral-factorized Positive-definite Curvature Learning for NN Training](https://mlsanity.com/paper/2502.06268/) (2025)
- [SVRG and Beyond via Posterior Correction](https://mlsanity.com/paper/2512.01930/) (2025)

## Abstract (arXiv)

The Bayesian learning rule is a natural-gradient variational inference method, which not only contains many existing learning algorithms as special cases but also enables the design of new algorithms. Unfortunately, when variational parameters lie in an open constraint set, the rule may not satisfy the constraint and requires line-searches which could slow down the algorithm. In this work, we address this issue for positive-definite constraints by proposing an improved rule that naturally handles the constraints. Our modification is obtained by using Riemannian gradient methods, and is valid when the approximation attains a block-coordinate natural parameterization (e.g., Gaussian distributions and their mixtures). We propose a principled way to derive Riemannian gradients and retractions from scratch. Our method outperforms existing methods without any significant increase in computation. Our work makes it easier to apply the rule in the presence of positive-definite constraints in parameter spaces.