# EigenVI: score-based variational inference with orthogonal function expansions arXiv: https://arxiv.org/abs/2410.24054 · HTML: https://mlsanity.com/paper/2410.24054/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 31 of its 54 references that are in the map. - First posted on arXiv 31 Oct 2024. ## Builds on (most cited first, 30 of 31) - [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015) - [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016) - [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016) - [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019) - [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016) - [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016) - [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018) - [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013) - [Automatic Differentiation Variational Inference](https://mlsanity.com/paper/1603.00788/) (2016) - [Multiplicative Normalizing Flows for Variational Bayesian Neural Networks](https://mlsanity.com/paper/1703.01961/) (2017) - [Sylvester Normalizing Flows for Variational Inference](https://mlsanity.com/paper/1803.05649/) (2018) - [Smooth Normalizing Flows](https://mlsanity.com/paper/2110.00351/) (2021) - [Subtractive Mixture Models via Squaring: Representation and Learning](https://mlsanity.com/paper/2310.00724/) (2023) - [Variational Boosting: Iteratively Refining Posterior Approximations](https://mlsanity.com/paper/1611.06585/) (2016) - [Advances in Black-Box VI: Normalizing Flows, Importance Weighting, and Optimization](https://mlsanity.com/paper/2006.10343/) (2020) - [Challenges and Opportunities in High-dimensional Variational Inference](https://mlsanity.com/paper/2103.01085/) (2021) - [Deep Directed Generative Models with Energy-Based Probability Estimation](https://mlsanity.com/paper/1606.03439/) (2016) - [Tensor-Train Density Estimation](https://mlsanity.com/paper/2108.00089/) (2021) - [Pathfinder: Parallel quasi-Newton variational inference](https://mlsanity.com/paper/2108.03782/) (2021) - [Boosting Variational Inference](https://mlsanity.com/paper/1611.05559/) (2016) - [Batch and match: black-box variational inference with a score-based divergence](https://mlsanity.com/paper/2402.14758/) (2024) - [Robust, Accurate Stochastic Optimization for Variational Inference](https://mlsanity.com/paper/2009.00666/) (2020) - [Semi-Implicit Variational Inference via Score Matching](https://mlsanity.com/paper/2308.10014/) (2023) - [Black Box Variational Inference with a Deterministic Objective: Faster, More Accurate, and Even More Black Box](https://mlsanity.com/paper/2304.05527/) (2023) - [Variational Inference with Gaussian Score Matching](https://mlsanity.com/paper/2307.07849/) (2023) - [Kernel Exponential Family Estimation via Doubly Dual Embedding](https://mlsanity.com/paper/1811.02228/) (2018) - [Variational Hamiltonian Monte Carlo via Score Matching](https://mlsanity.com/paper/1602.02219/) (2016) - [Boosting Black Box Variational Inference](https://mlsanity.com/paper/1806.02185/) (2018) - [Variational approximations using Fisher divergence](https://mlsanity.com/paper/1905.05284/) (2019) - [Energy-Inspired Models: Learning with Sampler-Induced Distributions](https://mlsanity.com/paper/1910.14265/) (2019) ## Built on by (most cited first, 11 of 11) - [Variational Inference with Mixtures of Isotropic Gaussians](https://mlsanity.com/paper/2506.13613/) (2025) - [How to Square Tensor Networks and Circuits Without Squaring Them](https://mlsanity.com/paper/2512.17090/) (2025) - [Towards Understanding Gradient Dynamics of the Sliced-Wasserstein Distance via Critical Point Analysis](https://mlsanity.com/paper/2502.06525/) (2025) - [Generalized Guarantees for Variational Inference in the Presence of Even and Elliptical Symmetry](https://mlsanity.com/paper/2511.01064/) (2025) - [On Faster Marginalization with Squared Circuits via Orthonormalization](https://mlsanity.com/paper/2412.07883/) (2024) - [Fisher meets Feynman: score-based variational inference with a product of experts](https://mlsanity.com/paper/2510.21598/) (2025) - [How to Approximate Inference with Subtractive Mixture Models](https://mlsanity.com/paper/2604.16714/) (2026) - [Scalable Expectation Estimation with Subtractive Mixture Models](https://mlsanity.com/paper/2503.21346/) (2025) - [On the Approximation Complexity of Matrix Product Operator (MPO) Born Machines](https://mlsanity.com/paper/2605.11471/) (2026) - [Large-scale empirical tuning and comparison of default optimizers for variational inference](https://mlsanity.com/paper/2606.07841/) (2026) - [QuanVI: Score-based Variational Inference via Quantum Maximally Mixed States](https://mlsanity.com/paper/2609.39164/) (2026) ## Abstract (arXiv) We develop EigenVI, an eigenvalue-based approach for black-box variational inference (BBVI). EigenVI constructs its variational approximations from orthogonal function expansions. For distributions over ℝD, the lowest order term in these expansions provides a Gaussian variational approximation, while higher-order terms provide a systematic way to model non-Gaussianity. These approximations are flexible enough to model complex distributions (multimodal, asymmetric), but they are simple enough that one can calculate their low-order moments and draw samples from them. EigenVI can also model other types of random variables (e.g., nonnegative, bounded) by constructing variational approximations from different families of orthogonal functions. Within these families, EigenVI computes the variational approximation that best matches the score function of the target distribution by minimizing a stochastic estimate of the Fisher divergence. Notably, this optimization reduces to solving a minimum eigenvalue problem, so that EigenVI effectively sidesteps the iterative gradient-based optimizations that are required for many other BBVI algorithms. (Gradient-based methods can be sensitive to learning rates, termination criteria, and other tunable hyperparameters.) We use EigenVI to approximate a variety of target distributions, including a benchmark suite of Bayesian models from posteriordb. On these distributions, we find that EigenVI is more accurate than existing methods for Gaussian BBVI.