mlsanityData to 14 Oct 2026

Information Geometry of Variational Bayes

arXiv 2509.15641 ↗First posted 19 Sep 2025cs.LG cs.AI stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

We highlight a fundamental connection between information geometry and variational Bayes (VB) and discuss its consequences for machine learning. Under certain conditions, a VB solution always requires estimation or computation of natural gradients. We show several consequences of this fact by using the natural-gradient descent algorithm of Khan and Rue (2023) called the Bayesian Learning Rule (BLR). These include (i) a simplification of Bayes' rule as addition of natural gradients, (ii) a generalization of quadratic surrogates used in gradient-based methods, and (iii) a large-scale implementation of VB algorithms for large language models. Neither the connection nor its consequences are new but we further emphasize the common origins of the two fields of information geometry and Bayes with a hope to facilitate more work at the intersection of the two fields.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 15 of its 36 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

15 of 36 references.

References · 15

  1. 2015Weight Uncertainty in Neural Networkscited by 554
  2. 2013Black Box Variational Inferencecited by 319
  3. 2013A General Framework for Updating Belief Distributionscited by 175
  4. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  5. 2019Learning with Fenchel-Young Lossescited by 119
  6. 2021The Bayesian Learning Rulecited by 99
  7. 2020Gradient Estimation with Stochastic Softmax Trickscited by 67
  8. 2017Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Modelscited by 65
  9. 2024Variational Learning is Effective for Large Deep Networkscited by 65
  10. 2019Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximationscited by 51
  11. 2018Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Modelscited by 38
  12. 2020Handling the Positive-Definite Constraint in the Bayesian Learning Rulecited by 31
3 more
  1. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  2. 2019Stein's Lemma for the Reparameterization Trick with Exponential Family Mixturescited by 18
  3. 2023The Lie-Group Bayesian Learning Rulecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Composing Non-Conjugate Factor Graphs with Closed-Form Variational Inferencecited by 1
  2. 2026Information Geometry of Message Passingcited by 0